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T.nanx  you,  Noel  (Sinners!,  and  good 
morning  to  you  all.  On  behalf  of  all  of 
us  at  NASA,  I'm  pleased  to  welcome  you  to 
this  most  important  symposium. 

Nearly  years  ago,  one  of  the  fathers 
of  cybernetics,  Norbert  Weiner,  described 
automation  as  a  "new  development  which 
has  unbounded  possibilities  for  good  and 
for  evil." 

Mr.  Weiner  was  right,  of  course.  New 
technologies  are,  in  themselves,  neutral. 
It  is  only  in  our  capabilities  and 
intentions  to  apply  them  that  tney  cake 
on  moral  significance.  The  real 
importance  of  advanced  technology  lies 
in  what  we  do  with  it.  1  am  pleased  that 
this  Symposium  will  help  us  to  focus  on 
applying  some  of  mankind's  most  advanced 
technologies  to  create  unbounded 
possibilities  for  good. 

As  we  prepare  to  enter  the  Space  Station 
era,  it  is  clearer  than  ever  that  the 
Station’s  development  will  also  drive 
the  development  of  automation,  robotics 
and  advanced  computing,  both  in  space  and 
on  Earth. 

Advances  in  these  technologies,  to  be 
stimulated  by  the  Space  Station  Program, 
will  benefit  the  United  States  in  many 
ways. 

Such  advances  will  increase  productivity 
in  space  for  commerce  and  science. 

They  will  al3o  increase  productivity 
throughout  the  United  States  economy, 
a3  the  new  technologies  are  transferred 
back  to  industries  on  Earth.  And  they 
will  help  to  preserve  United  States' 
leadership  both  in  space  and  at  the 
cutting  edge  of  technology  on  Earth. 

NASA  welcomes  the  challenge  of  developing 
a  highly  automated  and  autonomous  Space 
Station.  Indeed,  such  a  facility  will 
be  essential,  not  only  to  the  station's 
development,  but  to  its  evolution  and  to 
future  space  endeavors  through  the  end  of 
the  century  and  beyond. 

The  successful  use  of  3uch  technologies 
will  ensure  that  future  space  missions, 

K'  manned  or  unmanned,  will  be 

beneficial,  productive  ard  cost- 
effective.  NASA's  goal  in  developing 
t..ese  technologies  is  to  ensure  that 
we  achieve  the  be3t  mix  of  people  and 
machines  to  do  our  work  effectively 
and  efficiently. 


If  we  can  meet  that  goal,  we  will  enhance 
both  the  productivity  and  t re  safety  of 
tne  pioneers  who  will  be  living,  wording 
and  learning  in  space  permanently  in  just 
a  few  short  years.  Indeed,  tne  safety 
of  our  people  is  NASA's  highest  priority, 
because  people  are  our  most  precious 
resource . 

What  is  NASA  doing  to  achieve  that 
balanced  mix  of  people  and  machines 
so  essential  to  future  progress? 

We  have  developed  a  research  program 
designed  to  exploit  automat  ion  and 
robotics  technology  to  the  highest 
degree  possible  consistent  with  our 
resources.  The  research  is  carried 
out  by  our  Office  of  Aeronautics  and 
Space  Technology,  and  its  fruits  are 
transferred  to  tin  «  other  NASA  program 
offices;  that  of  Space  Flight,  of  the 
Space  Station  and  of  Space  Science  and 
Applications . 

Our  research  program  has  three 
objectives.  First,  we  aim  to 
decrease  the  cost  of  ground  control, 
and  ground  processing  and  checkout; 
second,  we  want  to  increase  the 
capability  and  flexibility  of  space 
operations.  Finally,  we  want  to 
increase  the  probability  of  mission 
success . 

Artificial  Intelligence  Technology  will 
be  used  to  reduce  the  size  of  the  ground 
control  contingent.  Telerobotics  will  be 
used  to  enable  increased  space  assembly, 
servicing  and  repair. 

In  the  long  ”un ,  we  expect  the  program 
will  have  extremely  beneficial  results. 

Its  goals  are  to  decrease  mission 
operations  manpc  *  -  hy  75  per  cent; 
to  replace  half  o > *. : *  extra-vehicular 

activities  with  t-  Doties;  and  to 
enable  remote  asse~  ,  servicing  and 
repair  through  telerobotics  in  both 
geosynchronous  Earth  orbit  and  in  polar 
orbit  for  the  Space  Shuttle,  the  Space 
Station's  Orbiting  Maneuver ing  Vehicle 
and  Flight  Telerobotics  Servicer  and 
other  orbiting  facilities. 

NASA's  work  is  designed  to  complement 
t  Ke  extensive  efforts  of  industry  and 
the  military  in  special  a  rear  '•hit 
several  study  groups  have  identified 
as  requiring  NASA  leadership. 


*  ’»  t  jT  or  is  declared  a  work  of 
V/Vo rr.men*  and  is  no*,  vjh 
~r;'  yr  nh*  T-roteo*  ion  in  tho 
’.'r.  i*ed 
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For  example,  one  area  in  whicn  NASA 
clearLy  .naa  taken  a  leadership  role  is 
that  of  traded  control.  As  you  know, 
this  is  t.ne  ability  to  switch  from  human 
to  autonomous  control  of  a  machine  grace¬ 
fully  and  smootniy  -  somethin?  like  using 
oruise  control  in  your  car.  Another  area 
;s  that  of  oper  3 1  or -ma  c.h  ;  n  e  interfaces. 

NASA  gained  this  expertise  oecause  if 
has  employed  automation  and  robotics  in 
oof n  manned  and  unmanned  missions  since 
the  beginning  of  t.ne  space  program.  Our 
unmanned  planetary  explorers  and  otner 
scientific  spacecraft  all  have  had  and 
continue  to  have  highly  automated 
systems.  And,  as  we  all  know,  the 
planned  Mercury  flights  were  so  automated 
that  the  astronauts  felt  they  would  be 
little  more  tnan  passengers  in  their 
capsules  . 

Our  heritage  in  robotics  was  highly 
visible  in  the  Viking  Mars  Lander, 
and  remains  so  in  the  Shuttle's  Remote 
Manipulator  System.  Clearly,  automation 
and  robotics  are  not  new  to  NASA. 

What  is  new  is  our  increased  focus  on 
advanced  automation  ana  robotics  tech¬ 
nology  in  recent  years.  This  new  3urge 
of  interest  and  activity  has  come  about 
because  of  two  related  government 
i n i t iat i ves . 

The  first  is  the  Space  Station  program, 
initiated  hy  President  Reagan  in  his  1984 
State  of  the  Union  Address.  The  second 
is  high  Congressional  interest.  This  was 
manifested  in  July  !  9  8  4  when  Congress 
passed  legislation,  now  known  a3  Public 
Law  g8-37l ,  which  directed  NASA  to  put 
the  best  brains  in  the  country  to  work 
to  identify  and  develop  space  station 
systems  which  advance  automation  and 
robotics  technologies,  not  only  on  the 
initial  Space  Station,  but  throughout 
its  evolution. 

That  same  public  law  spawned  the 
formation  of  two  groups.  The  first 
is  the  NASA's  Advanced  Technology 
Advisory  Committee,  a  group  of  NASA 
experts  formed  to  make  the  reports  the 
law  made  mandatory.  The  second  is  the 
Automation  and  Robotics  Panel,  composed 
of  non-NASA  experts  from  industry  govern¬ 
ment  and  academia.  Both  of  these  groups 
have  made  and  continue  to  make  valuable 
contributions  to  the  NASA  program. 

Their  advice  and  assistance  has  been 
invaluable  in  developing  our  agency-wide 
focused  and  augmented  automation  and 
robotics  program. 

We  are  developing  the  program,  as  we 
develop  any  major  program,  based  on 
strict  assignments  of  roles  and 
responsibilities  and  careful  p’anning 
a33ua.p*  "u.  Indeed,  we  had  ten  planning 
assumptions,  and  they  have  served  as  our 
Ten  Commandments  as  the  p-ng-am  t.t!:cc 
chape  . 
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Because  they  are  so  important,  '  would 
like  to  summarize  tnose  planning 
assumptions  now. 

The  first  is  that  tner e  will  oe  two  foci: 
felaroootics  and  system  autonomy,  jr 
expert  systems. 

Second,  eac.n  focus  will  nave  a  series  of 
ground  demonstrations  of  an  evolutionary 
testbed  to  show  increasing  capability  of 
integrated  technologies. 

Third,  there  will  be  a  core  technology 
program  to  develop  the  capabilities 
needed  to  enable  the  demonstration 
sequences  . 

Fourth,  the  resources  balance  will  be 
2/3  for  core  technology  and  1/3  for  the 
demonstration  sequences. 

Fifth,  any  flight  demons tr_t i cns  will  be 
funded  in  conjunction  with  the  NASA  User 
Offices,  that  is,  the  Office  of  the  Space 
Station  and  the  Office  of  Space  Flignt. 

Sixth,  each  of  the  two  ground 
demonstration  sequences  will  take  place 
at  a  NASA  Center,  but  not  necessarily  the 
same  center. 

Seventh,  the  core  technologies  will  be 
developed  at  various  sites. 

Eighth,  government,  industry  and 
university  research  will  be  leveraged  by 
using  HO  percent  of  the  program  funding 
to  sponsor  research  outside  of  NASA. 

Ninth,  for  each  focus,  the  technology 
development  will  begin  with  an  early 
demonstration  of  current  capabilities 
and  move  forward  in  aggressive,  but 
reasonable  increments,  leading  to  the 
development  of  a  revolutionary  technology 
for  both  automation  and  robotics. 

Finally,  arrangements  will  be  made  to 
link  research  centers,  user  centers, 

NASA  program  offices  and  appropriate 
universities  and  industries  into  working 
teams . 


Using  the  approach  I  have  just  outlined, 
NASA  i3  moving  vigorously  to  integrate 
application  of  advanced  automation  and 
robotics  technology  into  the  development 
of  the  Space  Station  and  into  the  Space 
Transportation  System  as  a  whole.  Our 
most  critical  needs  for  those  tech¬ 
nologies  for  the  Station  will  be  in 
the  initial  assembly  and  building  phase; 
for  ai'^nnosous  operations,  once  it  is 
assembled  and  built;  and  in  the  efficient 
servicing  of  the  Station,  payloads  and 
satellites . 


It  is  obvious  that  our  contractor's  — 

cooperation  is  vital  in  this  program. 

That  is  why  we  have  made  it  clear  to  - — 

t-wom  ('■■'♦■h  in  the  Phase  Z  dof  Initio,.  ~~J 
preliminary  design  phase  of  the  program, 

Coat 

d/or 

•>  t  a  t  ipuc  lai, 
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jn-j  as  we  approach  the  developmental 
psase,  i n a t  automation  and  robotics 
techno  logy  is  required  for  Pile  St/dee 
Station.  Indeed ,  wa  will  evaluate 
contractor  proposals  for  the  final  design 
and  fa  or i oat i on  pnase  of  tha  program  for 
p n e i r  responsiveness  to  automation  and 
r  :  o o :  .  j s  requiremer.ps  . 

[  am  p »a  sed  pr.  at  our  contractors  are 
coming  ..nro-gn  very  well.  Tr.ey 
.oenpifed  and  recommended  automation 
ana  r toot  ios  applications  during  Phase  B, 
and  have  amplified  those  plans  in 
innovative  and  skillful  ways  as  we 
prepare  for  development  in  Phases  C 
and'  3. 

Ponding  for  NASA's  automation  and 
robotics  research  and  technology  program 
is  growing  incrementally  every  year. 

It  has  gone  from  $10.2  million  at  the 
program's  inception  in  FY  1986  to  a 
proposed  $25.7  million  for  Fiscal  1988. 

I  am  pleased  to  report  that  funds  have 
been  committed  and  significant  progress 
has  been  made  in  developing  a  flight 
telerobot  servicer.  This  machine  will 
be  available  as  the  first  element  launch 
of  the  Space  Station  and  will  be  used  to 
3-jpport  assembly,  maintenance,  and 
servicing  of  the  station. 

Research  and  development  also  is  being 
conducted  throughout  NASA  and  by  its 
contractors  on  expert  systems  to  support 
Space  Station  operations  and  scientific 
research  and  manufacturing  in  space. 

The  university  community  is  on  board,  as 
well.  NASA  ha3  selected  the  University 
of  Wisconsin  to  establish  a  Center  for 
the  Commercial  Development  of  Space, 
specializing  in  space  automation  and 
robot i cs . 

As  we  move  forward  with  these  activities 
it  is  clear  that  potential  applications 
of  this  work  will  be  important  not  only 
for  the  Space  Station  program,  but  for 
the  Space  Transportation  System  as  a 
whole  and  for  industries  on  Earth. 

Indeed,  given  the  long-term  benefits  in 
greater  Space  Station  productivity  and 
in  major  productivity  improvements  that 
would  result  from  transfer  of  expert 
systems  to  industries  on  Earth,  many 
believe  that  NASA  should  make  a  greater 
investment  in  automation  and  robotics 
technology . 

For  example,  the  recent  report  of  the 
Automation  and  Robotics  Task  Force 
commissioned  by  NASA'3  Space  Systems  and 
Technology  Advisory  Committee  estimates 
that  the  program  is  underfunded  by  a 
factor  of  2  to  4.  The  Task  Force 
asserted  that  to  be  successful,  and  to 
do  all  that  NASA  should  be  doing  in  this 
promising  area,  funding  should  be 
increased  substantially. 


NASA  agrees  with  tr.at  assessment.  Art  I 
tan  only  say  fnat  if  tne  Congress  will 
g  i  ve  us  tne  resources  ,  we  will  do  tr.e 
oest  joo  that  can  oe  done. 

Our  great  adventure  in  space  has  hardly 
begun.  The  Space  Station  and  the 
missions  to  follow  will  cnallenge  our 
ingenuity  and  skills  to  the  fullest.  3ut 
from  this  effort  will  come  new  knowledge 
and  new  technological  breakthroughs  we 
can  hardly  imagine  today. 

Our  challenge  is  to  make  the  most  of  our 
opportunities  to  benefit  ourselves  and 
future  generations. 

La3t  month,  President  Reagan  spoke  to  the 
NASA  employees  on  the  first  anniversary 
of  the  Challenger  accident.  He  3aid: 
"Whether  it  i3  the  exploration  of  space 
or  the  applications  of  space  research 
here  at  home ,  the  future  to  which  you  are 
leading  us  is  bright,  the  challenge  whion 
you  are  shouldering  for  all  mankind  is 
one  that  we  cannot  turn  away  from." 

NASA  welcomes  that  challenge.  We  will 
never  turn  away  from  it.  We  will 
continue  to  push  ahead,  with  dedication 
and  resolve,  toward  a  brighter  future  for 
all  mankind. 

Thank  you  very  much. 
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Abstract 

T~e  U  S  Navy  has  a  woe  spectrum  of  applications  to 
wnicn  Ai  c3.o  be  applied,  including  manufacturing  and 
ogistics,  and  operational  applications  in  surface  ships, 
suomannes.  aircraft  and  space  applications  The  Navy, 
through  the  Office  of  Naval  Research  (ONR),  has 
suooorted  artificial  intelligence  research  from  its  earliest 
emergence  as  a  discipline,  and  continues  to  be  an 
moortant  contributor  to  university-based  artificial 
me  rgence  research  Increasingly  the  Navy  is 
conducting  applied  research  programs  within  its  own 
aboratories  with  the  goal  of  transitioning  the  technology 
,nto  service.  These  programs  cover  a  range  of 


application  areas,  many  of  which  are  closely  related  to 
civilian  problems,  but  often  they  exhibit  a  unique  military 
flavor  Applications  such  as  fault  diagnosis,  inspection, 
planning  aids  and  image  analysis  clearly  have  close 
counterparts  in  the  commercial  world.  Target 
classification,  battle  management,  naval  message 
processing  and  enemy  plan  recognition  are  unlike  their 
closest  commercial  counterparts  in  a  number  of 
respects,  particularly  when  operational  impact,  real-time 
operation  and  computing  power  limitations  are 
considered.  Navy  projects  in  several  of  these  areas 
have  matured  to  the  degree  that  full  scale  utilization  of 
the  technology  could  begin  within  three  to  five  years. 
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AND  THEMES  IN  INFORMATION  SCIENCE  AND  TETHNDLOCY 


:^ul  Ana  re  1 

'ofense  Advanced  Research  °roiects  Aaency 
Information  Science  and  Technology  Office 
Ar.inqtcn,  Virginia  JSS09-230B 


'  ■  ■'•'rut  i  no  is  in  extretrelv  iyn.iric  r  i  e  1  a ,  and  it 
w 1 1  ’  :T.ti-.ue  t  .*  nave  rna  ’cr  irract  our  society . 
'"■ere  ire  -.ever  *  I  excit  ir-i  sc  lent  i  f  i  ..-/techno  log :  c  i 1 
level 'prm.ts  :n  *he  fieli  today  -rat  promise  -o 
iccelerite  fur- ter  the  impact  of  icnnut mo  on 
!e*  ense  ir.d  'r.  ur  -  at  i^nal  ecnnrrr/ . 

Tue  technology  of  Very  r.arqe  Scale  In  too  rat. on 
(VLSI)  and  networking  has  spawned  several  novel 
"u  1 1 i processor  architectures.  These  architectures 
promise  to  r;ovide  the  very  hiqh  performance 
capabilities  that  are  needed  in  applications  such 
as  Vision,  Speech,  Complex  Svmbolic  Processing 
ml  raroe  Scientific  problems.  DARPA  has  a  maior 
effort  in  this  area  as  part  of  our  Strategic 
Computing  Program.  Results  so  far  are  very 
er.ccurag ing .  Much  remians  to  be  done,  especially 
:n  ittumnq  i  better  understanding  of  the 
capabilities  of  new  architectures  and  in 
developing  effective  wavs  of  applying  them  to  the 
solution  of  various  types  of  computat ionally- 
mtensive  problems.  Basically,  we  face  the 
challenge  of  developing  aopronriate  computat ional 
paradigms  for  naralleL  computing.  Another 
important  task  is  to  establish  better  links 
between  developments  in  conventional  (primarily 
numerical lv-or rented)  supercomputers  and  the  new 
explorations  in  highly  parallel  architectures. 

>.re  are  now  at  a  point  where  Artificial  Intelligence 
AI)  is  starting  to  have  noticable  impact  on  a 
number  of  industrial  and  defense  problems.  DARPA 
has  been  the  primary  source  of  support  for  AI 
research  over  the  last  twenty  years.  Scientific 
developments  in  AI  promise  to  elucidate  the  nature 
rf  intelligent  action  in  man  and  machine;  in 
addition,  thev  promise  to  bring  about  a  new 
veneration  of  machine  intelligence  technology  that 
will  extend  considerably  the  scope  and  power  of 
computing  that  will  be  available  to  various 
classes  of  users.  The  major  thrust  of  the  DARPA 
Strategic  Computing  Program  is  to  accelerate  in  a 
substantial  way  the  development  of  a  machine 
intelligence  technology  that  can  have  a  significant 
impact  on  defense  systems  and  that  can  strengthen 
our  industrial  capabilities.  Much  work  in  AI  is 
now  directed  to  the  exploration  of  methods  and 
svsteros  for  solving  very  complex  "real  life" 
nroblems.  In  parallel  with  these  efforts,  we  need 
Co  pursue  basic  research  on  problems  of  repre¬ 
sentation,  reasoning,  learning  and  discovery,  and 
on  frameworks  for  designing  and  implementing  A I 
systems . 

This  country,  and  the  DoD  in  particular,  has  an 
enormous  investment  in  software.  This  is  an  area 
where  better  methodologies,  tools  and  theories 
are  sorely  needed.  We  must  develop  much  more 
effective  ways  of  designing,  building  and  main¬ 
taining  software.  AI  methods  and  cost-effective, 
massive,  computer  power  will  certainly  help  us  in 
this  enterprise.  In  addition,  we  need  to  develop 
better  theoretical  underpinnings  and  more  powerful 
automated  processes  to  support  the  engineering  of 
complex,  testable  and  reliable  software. 


We  have  made  enormous  progress  in  ti.e  area  of 
computer  networking  and  distributed  computing .  we 
must  build  on  top  of  our  experience  with  ARPANET 
md  internetting  to  provide  a  more  convenient 
network  access  to  the  national  (and  international) 
scientific  and  engineering  conmunit les .  We  can 
expect  completely  new  patterns  of  collaborative 
research  and  engineering  to  develop  because  of  the 
growtn  and  maturation  of  computer  networking 
technology.  Experience  to  date  suggests  that  these 
developments  will  have  significant  impact  on  the 
productivity  of  professionals.  More  research  is 
needed  on  wide-band  networks  and  on  reconf igurable 
networks.  Also,  we  are  still  facing  difficult 
problems  in  the  design  and  use  of  distributed 
computing  systems. 

In  recent  years  there  has  been  substantial  proqres 
in  the  application  of  computers  to  design  and 
manufacturing.  This  is  an  enormously  promising 
area,  and  it  has  qreat  significance  for  the 
national  economy.  Most  of  the  ’work  so  far  has 
concentrated  on  the  use  of  computers  to  design 
computers  and  their  parts  (e.g.,  VLSI  design, 
digital  circuit  design,  system  conf iguration ) .  It 
is  becoming  apparent  that  many  of  the  current 
techniques  can  be  extended  to  other  types  of 
design,  including  the  design  of  mechanical  parts 
and  processes.  Computer  science  and  technology  is 
now  at  a  point  where  it  can  provide  the  intellec¬ 
tual  basis  and  the  tools  for  the  development  of  a 
science  of  design.  I  believe  that  the  entire  area 
of  exploring  computing  as  a  basis  for  assisting 
and  amplifying  industrial  productivity  is  in 
excellent  candidate  for  a  major  national 
initiative . 

Even  though  the  U.S.  pioneered  many  of  the  early 
developments  in  robotics,  it  is  now  lagging 
(relative  to  Japan)  in  the  production  and  use  of 
robotics.  We  now  have  many  of  the  elements  that 
are  needed  for  the  construction  of  advanced 
robots,  i.e.,  robots  that  can  effectively 
integrate  perception,  planning  and  action  in  the 
pursuit  of  given  goals.  At  DARPA  we  have  a  growing 
program  in  this  area,  including  the  Autonomous 
Land  Vehicle  project.  Hiis  is  another  area  that 
may  be  re  ly  for  a  major  national  thrust. 

There  has  been  progress  in  computer-aided 
instruction  and  training  in  the  last  decade  but 
nuch  more  needs  to  be  done  in  this  area.  By 
increasing  the  effectiveness  of  computer  systems 
as  adjuncts  to  instruction  we  can  improve  the 
human  potential  of  the  nation.  This  is  an 
extremely  important  objective.  We  need  additional 
research  in  this  area,  in  particular  work  on 
effective  and  adaptive  man-machine  interfaces. 
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We  are  mcvmq  to  a  point  where  the  presence 
computers  is  influencing  in  i  substantial  way  the 
ipproach  to  many  scie n tific  and  eng  ine e r i nq 
probl ems  .  Bas  •  c  paradigms  for  work,  m  physics, 
m  mathematics  and  in  other  areas  are  bemq 
reshaped  because  of  the  avaiiabilitv  <■  t'  vast  I v 
increased  ime.unts  of  computer  power  as  well  is  -r 
improved  neth-.  doK.-g  Les  for  harnessing  nw  computer 
power  to  the  needs  of  users.  It  is  important  r  r 
computer  science  and  technology  to  be  aware 
these  devel opmer.t s  and  to  try  to  understand  them, 
rhis  has  lirrpl  icat ions  in  education  in  romputer 
science  and  on  the  need  for  cross-discipl inarv 
research . 

"omputer  Science  is  continumq  to  stiff er  from  a 
manpower  shortage.  It  is  essential  to  build 
university  environments  that  encouraqe  an 
increased  production  of  graduate  students  in  the 
discipline.  This  has  implications  on  the 
continuity  of  research  support  and  the  avaiiabilitv 
of  up-to-date  computa t iona I  facilities  in 
academia . 

We  need  to  maintain  an  adequate  growth  rate  of 
computer  science  research  funding  in  the  nation. 

At  P>ARPA ,  computing  is  an  area  of  high  priority, 
and  it  is  being  funded  at  a  level  'which  is 
rouohly  one  quarter  of  the  agency’s  total  budget. 

It  would  be  desirable  to  develop  a  national  poll  ^ 
for  computer  science  research .  In  order  to  move 
towards  such  a  policy,  it  is  important  to 
increase  the  coordination  of  planning  in  computer 
science  among  government  agencies.  The  FCCSET 
committee  activities  are  a  move  in  the  right 
direction.  Also,  it  would  be  useful  to  establish 
a  forum  for  the  interchange  of  ideas  on  lrng- 
rancre  planning  with  part  icipat  ion  from  government, 
industry  and  academia.  The  creation  of  the 
Computer  Science  and  Technology  Board  bv  the 
National  Research  Council  is  an  important  step  in 
that  direction. 

'omputer  science  and  technology  are  in  a  healthv 
state  of  development  today.  There  are  many 
research  and  engineering  challenges  in  the  field; 
there  are  alro  many  opportunities.  Viewed  in  a 
broader  context,  the  computer  field  is  in  a 
pivotal  position  of  influence  vis-a-vis  all  other 
ureas  of  science  and  advanced  technology  in  the 
U.S.  It  is  essential  that  we  maintain  our 
momentum  in  advanced  computing  in  order  to 
ma inta in/ increase  the  chances  of  national 
leadership  in  other  key  areas  of  science  and 
technology,  such  as  Space. 
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t.  Introduction 

Robotics  is  wtiefe  Artificial  Intelligence  meets  the  real  world.  Al 
deals  with  symbols,  odes,  and  abstractions,  reasoning  about  concepts 
and  relationships  The  real  world.  In  contrast.  Is  tangible,  ful  ol 
exceptions  to  the  rules,  and  often  stubbornly  difficult  lo  reduce  to 
logical  expressions  Robots  must  span  that  gap  They  Hve  In  the  real 
world,  and  must  sense,  move,  and  manipulate  real  objects.  Vet  to  be 
intelligent,  they  must  also  reason  symbolically  The  gap  is  especially 
ptonounced  In  the  case  ol  outdoor  mobile  robots  The  outdoors  Is 
constantly  changing,  due  to  wind  In  trees,  changing  sun  positions, 
even  due  lo  a  robot's  own  tracks  from  previous  runs  And  mobility 
means  that  a  robot  Is  always  encountering  new  and  unexpected 
events  So  stalk:  models  or  preloaded  maps  are  inadequate  to 
represent  the  robot's  world 

The  loots  a  robot  uses  to  bridge  the  chasm  between  the  external 
world  and  Its  Internal  representation  Include  sensors,  image 
understanding  to  interpret  sensed  data  geometrical  reasoning,  and  a 
concept  of  time  and  ol  the  vehicles  motion  over  time  We  are 
studying  those  Issues  by  building  a  mobile  robot,  the  CMU  Navlab, 
and  giving  It  methods  ol  understanding  the  world  The  Navlab  has 
perception  routines  lor  understanding  color  video  Images  and  lor 
interpreting  range  data  CODGER,  our  ‘whiteboard",  which  was 
developed  for  the  Navlab  and  Its  smaller  cousin  the  Terregalor, 
handles  much  ol  the  modeling  ol  time  and  geometry  Our  architecture 
coordkiales  control  and  Intormatloo  (low  between  the  high-level 
symbo*c  processes  running  on  general  purpose  computers  and  the 
lower-level  control  running  on  dedicated  real-time  hardware  The 
system  but  It  from  these  tools  Is  now  capable  ot  driving  the  Navlab 
along  narrow  asphalt  paths  near  campus  while  avoiding  trees  and 
pausing  (or  loggers  that  get  In  Its  way 

This  report  describes  the  Navlab  [11)  and  the  software  we  have 
built  over  the  past  year  color  vision,  lor  finding  and  lo*owlng 
roads  (121.  30  perception,  lor  obstacle  avoidance  [4J;  and  the 
CODGER  whiteboard  (1 01 

2.  The  Navlab 

The  Navlab  ishort  lor  Navigation  Laboratory!  Is  a  van  converted  Into 
a  robot,  capable  ol  being  driven  conventionally  or  under  computer 
control  (figure  1 1  K  Is  self  contained,  carrying  Its  own  sensors, 
processors,  power,  and  even  researchers 

Our  previous  mob*e  robots,  such  as  the  Terregalor,  have  been 
rentable  workhorses  lor  small-scale  experiments  However,  we  have 
started  lo  outgrow  their  capabSttfes  As  we  began  to  experiment  wtth 


Figure  1 :  The  Na^ab 
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sensor  fusion,  smaller  robots  ran  out  ot  space  and  power  (or  multiple 
sensors  When  we  wanted  to  expand  our  test  areas,  communications 
to  a  remote  computer  In  (he  lab  became  more  difficult  And  as  the 
experiments  became  more  sophisticated,  we  lound  It  more  productive 
(or  the  experimenters  lo  test  or  debug  new  programs  near  or  in  the 
vehicle,  Instead  In  a  remotely  located  laboratory 

Ncvtab  Is  based  on  a  commercial  van  chassis,  with  hydrau«c  drive 
and  electric  steering  Computers  can  steer  and  drtve  the  van  by 
electric  and  hydraulic  servos,  or  a  human  driver  can  take  over  control 
it  necessary  It  Is  even  licensed  In  Peonsytvanla.  so  we  can  drtve  it  by 
hand  to  our  remote  test  sites  Once  there,  the  driver  pulls  a  switch 
and  an  onboard  microprocessor  assumes  control  of  the  steering  and 
drtve.  The  Navtab  has  room  tor  researchers  and  computers  on  board, 
and  has  enough  power  and  space  lor  al  our  existing  and  planned 
sensors.  This  gets  the  resoarchers  dose  to  the  experiments,  and 
eHminales  the  need  lor  video  and  digital  communications  with  remote 
computers. 

Features  ol  the  Navlab  Include 

•  Onboard  computers.  We  have  Hve  computer  racks,  one 
lor  low-level  controllers  and  power  smoothing,  one  lor 
video  distribution.  VCRs,  communications  and 
miscellaneous  equipment,  two  racks  lor  general-purpose 
processors  (currently  Sun  workstations),  and  one  tor  a 
Warp  processor 

•  Onboard  r»M archers  There  Is  always  a  safety  driver  In 
the  driver's  seat.  There  13  room  lor  lour  researchers  In 
the  back,  wflh  a  terminal  or  workstation  (or  each  An 
overhead  shelf  holds  video  monitors  and  additional 
terminals.  The  researchers  car  monitor  both  Ihelr 
programs  and  the  vehicle  s  motion 

•  Onboard  power.  The  Navlab  carries  two  5500  Watt 
generators,  plus  power  conditioning  and  battery  backup 
tor  critical  components 

•  Onboard  aenaors.  Above  the  cab  Is  a  pan  mount 
carrying  our  laser  scanner  and  a  mounting  ral  lor  a  color 
TV  camera  There  wfl  eventualy  be  a  separate  pan/tlH 
mount  tor  stereo  cameras 

•  Evolving  ControHar.  The  flrst  computer  controller  tor  the 
Navlab  Is  adequate  tor  our  current  needs  It  wit  evolve  to 
do  smoother  motion  control,  and  to  interface  to  an  Inert! al 
guidance  system  and  possfcty  to  GPS  sateWe 
navigation  The  controler  wit  also  watch  vital  signs  such 
as  computer  temperature  and  vehicle  hydraulc  pressure 

3.  Color  Vision 

The  Navtab  uses  color  vision,  specifically  mul#-dass  adaptive  color 
classification,  to  find  and  (dow  roads.  Image  points  are  class  Hied  Into 
"road"  or  "non-road"  prtndpaly  on  the  basis  of  their  color.  Since  the 
road  Is  not  8  uniform  color,  color  classification  has  to  have  more  than 
one  road  model,  or  class  and  more  than  one  non-road  class  And 
since  condWons  change  from  time  to  time  and  from  place  to  place 
over  the  test  course,  the  colors  have  to  adapt.  Once  the  Image  is 
classified,  fie  road  Is  found  wMh  an  area-based  voting  technique  that 
finds  the  most  fcety  location  tor  the  road  In  the  Image. 

3.1  Vlalon  Prlrtclplee  for  the  Real  Work! 

We  based  the  development  ot  our  vision  on  the  lot  lowing  principles 

Aaauma  variation  and  change.  On  sunny  days,  there  are 
shadowed  areas.  sunH  areas,  and  patches  wtth  dappled  sunlghl  On 
rainy  days,  there  are  dry  patches  and  wet  patches  Some  days,  there 
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are  wet  and  dry  and  sunny  and  shadowed  all  in  Ihe  same  image  The 
road  has  dean  spots  and  other  places  covered  with  leaves  or  with 
drips  ot  our  own  hydraulic  (Hjtd  And  as  the  sun  goes  behind  a  doud 
or  as  the  vehicle  turns,  lighting  conditions  change  This  means  that 
we  need  to  have  more  than  one  road  color  class  and  more  than  one 
non-road  class  that  Ihose  Hasses  need  lo  adapt  to  changing 
conditions,  and  that  we  need  to  process  images  frequently  so  the 
change  from  one  image  to  the  next  will  pe  moderate 

Use  tew  geometric  parameters  A  complete  description  cl  the 
mad's  shape  in  an  image  can  be  Quite  como lex  The  road  can  bend 
gently  or  turn  abruptly,  can  vary  in  width  and  can  ao  up  or  down  hill 
However  the  more  parameters  there  are.  the  greater  the  chance  ol 
mror  m  finding  those  parameters  Small  mlsclassltlcatlons  in  an 
mage  could  give  use  to  lairiv  large  errors  m  perceived  road  geometry 
'T  uiiliermcre  ■'  all  Ihe  road  parameter  tan  vary  (here  ambiguous 
nierprelatlons  does  the  road  actually  rise,  or  does  it  instead  get 
wider  as  >t  goes  '  vVe  chose  to  describe  the  road  wtth  only  two  Iree 
parameters  its  orientation  and  its  distance  from  the  vehicle  Road 
width  is  fi«ed,  we  assume  a  Hat  world,  and  we  decree  that  the  road  Is 
straight  While  none  ol  those  assumptions  are  true  over  a  long  stretch 
ot  Hie  load,  they  are  nearly  true  within  any  one  image  And  the  errors 
in  road  position  that  come  from  ihose  oversimplifications  are  balanced 
by  'ess  chance  ol  bad  Interpretations.  11  there  are  a  lew  pixels 
incorrectly  classified  as  road,  the  worst  our  method  will  do  Is  lo  lind  a 
slightly  incorrect  road  A  method  that  tnes  to  tit  more  parameters,  on 
the  other  hand,  may  come  up  with  the  perfect  Interpretation  ol  part  ot 
me  road,  but  could  find  an  abrupt  turn  or  sudden  slope  near  the  bad 
pixels. 

Work  In  the  Image.  The  road  can  either  be  found  by  protecting  the 
road  shape  into  the  image  and  searching  in  image  coordinates,  or  by 
back  projecting  the  image  onto  Ihe  ground  and  searching  In  world 
coordinates  The  problem  with  the  latter  approach  comes  in  projecting 
the  image  onto  an  evenly  spaced  grid  In  ihe  world  The  points  on  the 
world  grid  close  to  the  vehicle  correspond  lo  a  big  area  In  the  lower 
part  of  the  Image,  points  farther  away  may  correspond  to  one  or  a  few 
pixels  near  the  top  So  either  there  has  to  be  a  complex  weighting 
scheme,  or  some  Image  pixels  (those  at  the  top,  that  pro|ed  to  distant 
world  points)  wHI  have  more  weight  than  other  (lower)  points  A  lew 
noisy  pixels  can  have  a  big  or  a  small  effect,  depending  on  where  In 
the  image  they  lie  On  the  other  hand,  working  directly  In  the  image 
makes  it  much  easier  to  weight  all  pixels  evenly  We  can  dlrectty 
search  lor  the  road  shape  that  has  the  greatest  number  of  road  pixels 
and  the  least  non-road  pixels  Moreover  projecting  a  road  shape  is 
much  more  efficient  than  back  projecting  all  the  image  pixels 

Calibrate  directly.  A  complete  description  ol  a  camera  has  tc 
indude  Hs  position  and  orien<a1ion  In  space;  Its  local  length  and 
aspect  ratio  lens  effects  such  as  fish-eye  distortion;  and  non- 
llnearttles  in  the  optics  or  sensor  The  general  calibration  problem  ol 
trying  lo  measure  each  ol  these  variables  Is  very  difficult  It  Is  much 
easier  and  more  accurate,  to  calibrate  Ihe  whole  system  rather  than 
trying  lo  tease  apart  the  Individual  parameters  The  easiest  method  Is 
to  take  a  picture  ol  a  known  object  and  build  a  lookup  lable  that 
relates  each  world  point  to  an  image  pixel  and  vice  versa.  Projecting 
and  back  projecting  are  simply  done  by  table  lookup,  or  table  lookup 
lor  dose  by  values  wtth  simple  Interpolations  Such  a  table  is 
straightforward  to  bulk)  and  provides  good  accuracy,  and  there  are  no 
instabilities  In  the  calculations 

uuistde  constraints.  Even  without  a  map  «l  our  lest  course 
or  an  expensive  Inertial  navigation  system,  we  know  approximately 
where  the  road  should  be  based  on  the  previous  image  and  on  vehicle 
motion  Our  'whiteboard'  can  predkt  where  the  road  should  appear  It 
the  road  were  straight  and  the  vehicle  navigation  were  period 
Adding  a  suitable  margin  lor  curved  roads  and  sloppy  navigation  still 
gives  useful  limits  on  where  In  the  Image  to  look  lor  Ihe  road 

Test.  We  tried  to  run  our  VCR  every  day  we  look  the  vehicle  out.  to 
colled  Images  under  as  many  conditions  as  possible  We  had  sunny 
days,  cloudy  days,  ralnv  days,  leaves  on  trees,  leaves  turning  color, 
leaves  lalHng,  early  morning,  noon,  after  dusk,  even  a  partial  solar 
edtpse  Strategies  that  worked  well  on  one  set  ol  Images  dk)  not 
atways  work  on  ihe  others  Our  philosophy  was  lo  colled  Ihe  toughest 
images  trom  those  sets  We  ran  our  best  algorithms  and  printed  the 
dasslflcatlon  results,  changed  parameters  or  algortlhms,  reran  the 
data  set.  and  compared  results  This  gave  us  the  best  chance  ol 
being  methodical  and  ol  not  kitrorkjdng  new  bugs  as  we  wenl  When 
the  'mage  processing  worked  to  our  satlsfadlon,  we  ran  simulations  In 
the  lab  that  Induded  the  whiteboard,  range  processing,  path  planning. 


and  a  -ebicte  simulator  with  .tsic.i  process  eg  sirred  maqes  and 
interacting  with  the  rest  ot  the  system  When  ihe  simulations  worked 
in  the  lab.  we  moved  Ihem  to  Ihe  vehide  Only  after  ihe  simulatloc.3 
worked  on  the  vehlde  s  computers,  and  wo  were  sure  that  a* 
necessary  software  was  on  ttie  van.  dk)  we  go  into  the  Held  lor  real 
tests  Even  then,  everything  dk)nt  work,  but  there  were  many  tewer 
bugs  than  there  would  have  been  without  the  simulations  and  tests 

3.2  Road  Following  Algorithm 


Figure  2:  Original  Image 

We  followed  Ihose  pdndples  In  building  and  tuning  adaptive  color 
dassificatlon  lor  following  roads  Our  algorithm  Involves  three  stages. 

1  Classify  each  pixel 

2  Use  Ihe  results  of  dasslflcatlon  to  vole  lor  Ihe  best-fit 
road  position 

3  Colled  new  color  statistics  based  on  the  detected  road 
and  non  road  regions 

Pixel  dasslflcattoo  Is  done  by  standard  pattern  dasslflcatlon  Each 
dass  Is  represented  by  means,  variances.  and  covariances  ol  red, 
green,  and  blue  values,  and  by  Its  a  pnorl  HkeHhocd  based  on 
expected  fraction  ol  pixels  In  that  dass  For  each  pixel,  calculating  Ihe 
dass  to  which  It  most  likely  belongs  Involves  finding  how  far  the  pixel's 
values  lie  trom  the  mean  ol  each  dass.  where  distance  Is  measured  ki 
standard  deviations  ol  that  dass  Figures  3  and  4  show  how  each 
pixel  Is  classified  and  how  well  It  matches 

Once  each  point  has  been  dassHled.  we  have  lo  find  ihe  most  nvefy 
location  ot  the  road  We  assume  the  road  Is  locally  straight,  and  cal 
be  described  by  two  parameters  (figure  5): 

1  The  Image  column  ol  Ihe  road's  'vanishing  point'  where 
Ihe  road  intercepts  the  horizon  This  gives  Ihe  road  $ 
dkectlon  relative  to  Ihe  vehicle 

2  The  orientation  of  the  road  In  Ihe  Image,  which  gives 
now  far  the  vehicle  Is  to  the  right  or  left  ot  the  centerline 

We  set  up  a  two  dimensional  parameter  space  with  Intercept  as  on* 
dimension  and  orientation  as  the  other  Each  point  dassilled  as  road 
votes  tor  at  road  intercept  1  orientation  combinations  to  which  It  could 
belong,  as  shown  m  figure  6  The  orientation  /  Intercept  pair  that 
receives  the  most  voles  Is  the  one  that  contains  the  most  road  points, 
and  Is  reported  as  the  road  (figures  7  wid  8). 

Once  the  road  has  been  foend  in  an  Image,  the  color  statistics  we 
recalculated  lor  each  dass  (figure  9)  The  updated  color  statistics  wil 
gradually  change  as  Ihe  vehicle  moves  Into  a  (Afferent  color  road,  or 
as  fighting  conditions  change,  or  as  the  color  ol  (he  surrounding  grass, 
dirt,  and  trees  vary.  As  long  as  the  processing  lime  per  Image  Is  low 
enough  that  there  Is  a  Iwge  overlap  between  images.  Ihe  statistics 
adapt  as  the  vehicle  moves.  The  road  Is  picked  out  by  hand  In  the 
first  Image  Thereafter,  the  process  Is  automatic,  using  the 
segmentation  trom  each  image  to  calculate  color  statistics  lor  the  next. 
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Figure  3:  Segmented  Image  Color  and  texture  cues  are  used  to 
label  points  below  the  horizon  into  two  road  and  two  offroad  classes 
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Flgur*  8:  The  point  from  figure  5  would  vote  for  these  orientation  , 
intercept  values  In  the  parameter  space. 


Flgur*  5:  A  point  dammed  as  road  could  be  a  part  of  roads  with 
oNterent  orientations  and  vanishing  points. 

There  are  several  variations  on  this  basic  theme.  One  vwtatlon  is 
to  smooth  the  Images  first.  This  throws  out  outliers,  and  tightens  the 
road  and  nonroad  cfrjsters.  Another  Is  to  have  more  than  one  class 
or  road  and  lor  non-road,  lor  Instance  era  tor  wet  road  aid  one  for 
r oL  Jf?°«_5r  shactow3  and  one  for  sun.  ff  we  knew  where  the  wet 
’W9'  eoM  colec*  5eParB*'  «** 
amafTL  dassw  Slnc»  *e  don't  know  where  those 

too  nan  5tat,sHc5  by  starling  with  one  class  tor  the 

stotlsto  and  °°*  ,or  *"  bo”om-  UstnQ  cokx 

nestles  from  those  regions,  we  redassdy  the  road  pixels  In  the  same 


Flgura  7:  Votes  tor  best  road  orientation  and  Intercept,  and  point 
with  most  votes. 


Figure  4:  Road  probability  image  The  pixels  that  best  match 
typical  road  colors  are  displayed  brightest 


Flgura  8:  Detected  road. 

Image,  and  rocalaSale  statistics  based  on  this  reclassification  We 
loop  through  the  dassfry  -  update  cyde  several  times  tor  each  Image 
As  long  as  the  first  two  point  sets  have  dltlerent  distributions,  the 
resting  classes  wa  move  towards  separate,  tighter  dusters. 

Other  variations  change  the  voting  tor  best  road.  Besides  adding 
VO<*?  road  P***®'  *•  subtract  v<*«  for  non-road  points.  Votes  are 
weighted  according  to  how  we#  each  point  matches  road  or  nonroad 
classes. 

There  are  otter  does  In  an  Image  besides  color,  lor  Instance  vtsud 
texture.  Roads  tend  to  be  smooth,  with  less  high-frequency  variation 
than  grass  or  leaves,  as  shown  In  figure  10.  Our  first  texture  algorithm 
ran  a  Robert's  gradient  operator,  Ihresholded  the  result,  sex)  counted 
the  number  of  pixels  above  threshold  In  each  16  by  16  block.  This 
me*>od  fafis  to  deal  adequately  wtth  shadow  edges,  to  which  Robert's 
operator  responds  strongly,  or  with  the  Interiors  of  shadows  In  which 
the  absolute  values  of  gradtents  are  small  because  the  pixel  values 

are  small.  The  shadow  edge  problem  can  be  hwidted  by  dividing  the 
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Figure  9  "pdaring  mag  and  n0nroad  model  colors  'eav-vg  a 
porv  :?ne  around  me  delected  road  region 


Figure  10-  2oomed  picture  of  road  nonroad  boundary  The  road 
jr  left j  rs  murrh  less  lectured  man  the  grass  'at  right) 

3  3  Implementation  and  Results 

■  (-  pest  'ombinallon  we  nave  lound  Is  to  nave  two  road  "desses 
i.-.g  rwo  non  road  desses  and  to  smooth  the  image  wttti  a  tb  by  16 
..waging  sner  The  two  road  classes  handle  sunny  and  shaded  road 
r.  right  lays  or  wet  and  dry  patches  "-der  overcast  conditions 
’  l-n  rwo  non  road  classes  often  converge  on  frees  and  grass,  or  grass 
md  dr.  'eaves  Filtering  (tie  images  by  averaging  reduces  the  abaci 
,i  rachs  m  the  'sidewalk  ar>d  other  scene  anomalies  and  also 
months  nil  'inter  aberrations  from  misaligned  earners  guns  and  Irom 
.*sn  m  the  digitizers  and  color  splitter 

•tie  'e.ture  mlormalton  could  be  used  as  a  loiirth  element  in  fhe 
iassifir;alion  along  with  red  green  and  bbte  We  lound  ledum  used 
way  to  he  less  *»aadie  than  :o*or  instead  we  heal  le  'll  ire  as  a 
v.yaraie  'eamre  with  toed  sfanstlrs  mean  and  variance  tor  road  and 


.-.npr« ag !  jve.  o  y  C’ls*  cr*..-pscmg  c’age  we  v  -  it-  -  ■  yt  uvl 
ronrnad  probabilities  based  in  :cior  ird  cepar  ste*. v  saiitd  a  ’vci.ru 
A’e  men  combine  tbe  'wo  with  'clot  weighted  "u*  h  "vre  ’lea.il-,  out 
•  e.pjre  dlscnminating  in  borderline  cases 

•i„s  algorithm  ,ioriecttv  aassifles  u.er  >6’-  -f  ‘m  '  ’manes  r 
■he  Mb  .Vlien  we  run  yn  »he  Ul.l'ib  wah  predirtinns  i  wt-*-re  e  e 

• .  ad  should  appeal  cur  dilute  rate  is  -or,  lose  to  "  r,,e  :.j  aswnal 
remaining  problems  ;ome  from  n aving  mo  many  iea.es  in  'he  read 
i  me  color  statistics  ot  one  road  class  and  o(  nonroao  are  ,»r,  dose 
r.’ot  every  mage  'S  cfassit'ed  perfectly  but  almosl  a'l  are  good  enough 
■  r  navigation  We  leave  a  50  pi«ei  ‘safety  :one‘  about  two  'eet  .-n 
:tie  ground  near  Ibe  .ebidei  along  ibe  road  border  tba'  is  noi  used  m 
updating  color  statistics,  lo  keep  Irom  being  jrawn  ott  bv  'man 
mistakes  or  curving  roads 


Figure  11:  Low  resolution  texture  image  yhe  brigbier  Mocks  are 
mage  areas  with  more  visual  texture 

There  is  no  need  lor  complelo  geometric  -alibraWon  The  vision 
aigortihms  calculate  tbe  toads  shape  iroad  width  and  location  ot  'he 
hotizoru  irom  the  llrst  training  image  We  also  lake  two  calihr anno 
pictures,  with  a  meter  slick  placed  perpendicular  to  the  vehicle  3  and 
12  meters  In  front  Then,  during  the  run  given  ibe  centerline  a 
delected  road  in  image  coordinates,  it  is  easy  lo  get  the  <  position  oi 
the  road  at  8  and  12  meters,  and  then  to  calculate  the  vehicles 
pos'lkgn  on  the  road 

This  algorithm  runs  in  about  10  seconds  per  image  on  a  dedtcatiwi 
Sun  3.160  using  480  by  512  pixel  images  'educed  lo  30  rows  by  32 
columns  We  currenttv  process  a  new  image  every  4  meters  which 
gives  about  three  lourths  ot  an  Image  overlap  between  images  T  on 
seconds  is  last  enough  to  balance  the  msi  ol  Ihe  system  hut  Is  skew 
enough  that  clouds  can  come  and  go  and  Hghfing  conditions  change 
between  images  We  plan  to  png  this  algorithm  lo  the  Warp  CMU  s 
experimental  high  speed  processor  We  hope  to  process  an  mage 
per  second,  and  to  use  higher  resolution 

4.  3D  Perception 

Our  obstacle  detection  starls  with  direct  range  perception  using  an 
ERIM  scanning  laser  rangefinder  Our  EPIM  produces  every  ball 
second,  an  image  conlalnlng  64  rows  by  256  columns  ol  range  vafcjes 
■  see  figure  12)  The  scanner  measures  tbe  phase  dftlerence  between 
an  amplitude- modulated  laser  and  Its  rotlectton  horn  a  large!  object 
which  in  turn  provides  ihe  distance  between  ibe  target  object  and  ihe 
scanner  The  scanner  produces  a  dense  range  Image  by  using  two 
deflecting  mirrors,  one  for  the  horizontal  scan  Hnes  and  one  lor  vertical 
motion  between  scans  The  vofcjme  scanned  Is  80  degrees  wide  and 
30  high  The  rtnge  at  each  pixel  Is  discretized  over  256  levels  irom 
zero  to  64  feet 


anexxie  distance,  nearer  points  are  painted  darker 


Out  range  processing  begins  by  smoothing  the  data  and  undoing 
the  peculiarities  ol  Ibe  ranging  geometry  The  "ambiguity  intervals", 
where  range  values  wrap  around  Irom  255  to  0  are  de'octed  and 
u.iloWed  Two  other  undersIraNe  effects  are  removed  by  the  same 
ulqorithm  The  first  one  is  the  presence  ot  mixed  points  at  the  edge  of 
an  object  The  second  Is  that  a  measurement  from  a  surface  such  as 
.•<  .er  glass.  o<  glossy  pigments  is  meaningless  In  both  cases,  the 
resulting  points  are  m  regions  Hmited  by  considerable  Jumps  in  range 
and  can  iheretore  be  removed  by  the  same  algorithm  We  then 
transform  ihe  values  from  angle-angle-range.  In  scanner  coordinates 
to  <y-z  locations  rtiese  30  points  are  Ihe  basts  for  all  further 
processing 

We  have  two  main  processing  modes  obstacle  defection  and 
terrain  analvsis  Obstacle  defection  starts  by  calculating  surface 
normals  from  ihe  x-y-z  points  Flat,  traversable  surfaces  will  have 
vertical  surface  normals  Obstacles  will  have  surface  patches  with 
normals  pointed  in  other  directions  This  analysis  is  relatively  last, 
running  in  aboul  5  seconds  on  a  Sun  3/75,  and  is  adequate  tor 
smooth  terrain  with  discrete  obstacles 

Simple  ob slade  maps  are  not  sufficient  lor  detailed  analysts  For 
greater  accuracy  we  do  more  careful  terrain  analysts  and  combine 
sequences  ol  images  corresponding  to  overlapping  parts  of  the 
environment  into  an  extended  obstade  map  The  terrain  analysis 
algorithm  first  attempts  to  find  groups  ot  points  that  belong  to  the 
same  surface,  and  ihen  uses  these  groups  as  seeds  for  the  region 
growing  phase  Each  group  Is  expanded  Into  a  smooth  connected 
surface  patch  The  smoothness  of  a  patch  is  evaluated  by  fitting  a 
surface  plane  or  quadric  In  addition,  surface  discontinuities  are  used 
to  limit  Ibe  region  growing  phase.  The  complete  algorithm  is 

1  Edges  Extrad  surface  dtscontinutttes.  pixels  with  high 
lumps  in  x-y-z 

2  Clustering  Find  dusters  In  Ihe  space  of  surface 
normals  and  identify  the  corresponding  regions  In  Ihe 
original  Image 

3  Fleglon  growing  Expand  each  region  untH  the  fitting 
error  is  larger  than  a  given  threshold  The  expansion 
proceeds  by  iteratively  adding  Ibe  ooint  ol  the  region 
boundary  that  adds  the  minimum  fitting  error 

The  dustering  step  Is  designed  so  that  other  attributes  such  as 
color  or  curvature  can  also  be  used  to  find  poteottal  regions  on  the 
ohjed  The  primitive  surface  used  to  compute  the  fitting  error  can  be 
either  a  plane  or  a  quadric  surface  The  decision  Is  based  on  the  size 
ot  the  region 

Obstade  detection  works  at  longer  range  than  terrain  analysis 
When  the  scanner  Is  looking  at  distant  objects.  It  has  a  very  shallow 
depression  angle  Adjacent  scanlines,  separated  by  t/2  degree  In  the 
range  image,  can  strike  the  ground  at  widely  different  points.  And 
since  the  grazing  angle  Is  shallow,  very  little  ol  Ihe  emitted  laser 
energy  returns  to  the  sensor,  producing  noisy  pixels.  Noisy  range 
values  widely  spaced,  make  It  difficult  to  do  detailed  analysis  ol  fiat 
lerraln  But  a  vertical  obstacle,  such  as  a  tree,  shows  up  much  better 
in  the  range  data  Pixels  (rom  neighboring  scanlines  Ia0  more  dosefy 
together,  and  with  a  more  nearly  perpendicular  surface  the  returned 
signal  Is  stronger  and  the  data  Is  cleaner.  So  tt  Is  much  easier  (or 
obstade  detedlon  to  find  obstacles  than  tor  terrain  analysis  to  certify 
that  a  patch  ot  ground  Is  smooth  and  level 

There  are  cases  in  which  neither  video  nor  range  alone  provide 
enough  information,  where  we  have  lo  do  data  fusion  to  determine 
mobility  or  recognize  an  object  One  such  example  occurs  In 
navigating  the  smaller  Terregator  vehicle  around  campus  sidewalks. 
At  one  spot,  a  sidewalk  goes  up  a  flight  ot  stairs  and  a  bicycle  path 
curves  around  Video  alone  has  a  tough  lime  distinguishing  between 
the  cement  staks  and  the  cement  bicycle  peth.  Flange  data  cannot 
tell  the  difference  between  the  smooth  rise  of  the  grassy  Nl  and  the 
smooth  btcyde  ramp.  The  onty  way  to  conectty  Identify  the  sale 
vehicle  path  is  lo  use  both  kinds  of  data. 

We  slarl  by  fusing  ihe  data  at  Ihe  pixel  level.  For  each  range  point, 
we  find  the  corresponding  pixel  In  the  video  Image  We  produce  a 
painted  range  Irrwge,  in  which  each  pixel  Is  a  (red,  green,  bkje.  x,  y,  z) 
6-vector  Then  we  can  run  our  standard  range  segmentation  and 
color  segmentation  programs,  producing  regions  of  smooth  range  or 
constant  color  For  Ihe  stairs  In  particular,  we  have  a  special-purpose 
step  detection  program  that  knows  about  vertical  and  horizontal 
planes  and  how  they  are  related  In  typical  stairs  It  Is  easy  to 


combine  ihe  regions  from  these  separate  processes,  snce  they  are  a* 
in  Ihe  same  coordinates  of  the  painted  range  mage  The  final  result 
is  a  smooth  concrete  region,  Ir.  which  It  Is  sale  lo  drive,  and  a  positive 
identification  and  3D  location  ol  the  stairs,  (or  updating  the  vehicle 

position 

5.  System  Building 

Besides  the  problems  in  building  the  various  modules  lor  road 
Porting,  obcfcctc  patn  planrang,  and  so  torth.  there  are  also 

a  set  ol  issues  related  to  building  a  coherent  system  We  have  buitl  a 
layered  system  with  CODGER  providing  tools  and  services,  and  an 
architecture  on  top  setting  conventions  lor  control  arid  data  Mow 

5.1  Blackboards  and  Whiteboards 

The  program  organization  ol  the  NAVLAB  software  is  shown  In 
figure  13  Each  ot  the  major  boxes  represents  a  separately  running 
program  The  central  database,  cafied  the  Local  Map,  is  managed  by 
a  program  known  as  the  Local  Map  BuSder  (LM8).  Each  module 
stores  and  retrieves  information  in  the  database  through  a  set  ot 
subroutines  called  the  LMB  Interlace  which  handle  all  communication 
and  synchronization  with  the  LMB  tt  a  module  resides  on  a  different 
processor  than  Ihe  LMB.  the  LMB  and  LMB  Interface  wifi  transparently 
handle  tire  network  communication  The  Local  Map.  LMB  and  LMB 
interlace  together  comprise  the  CODGER  (Communications 
Database  with  GEometric  Reasoning)  system 


Figure  13:  Navtab  software  architecture 

The  overafi  system  structure — a  central  database,  a  pool  ol 
knowledge-intensive  mockites,  and  a  database  manager  that 
synchronizes  the  modules — Is  characteristic  of  a  tracfittonal  blackboard 
system  Such  a  system  Is  called  "heferarchlcar  because  the 
knowledge  Is  scattered  among  a  set  of  modules  that  have  access  to 
data  at  all  levels  ol  the  database  (1  e.,  low  level  perceptual  processing 
ranging  up  to  high-level  mission  plans)  and  may  post  their  findings  on 
any  level  of  the  database;  In  general,  heterarchical  syslems  Impose  de 
lacto  structuring  of  the  Information  Bow  among  the  modules  ot  the 
system.  In  a  traditional  blackboard,  there  Is  a  single  flow  of  control 
managed  by  the  database  (or  blackboard)  manager.  The  modules  are 
subroutines,  each  with  a  pre  determined  precondMon  (pattern  ot  data) 
that  must  be  satisfied  before  trat  module  can  be  executed.  The 
manager  keeps  a  list  ot  which  modules  are  ready  to  execute,  and  In  Ns 
central  loop  It  selects  one  module,  executes  tt.  and  adds  *o  Ns  ready 
tst  any  new  modules  whose  precondWona  become  satisfied  by  the 
currently  executing  module.  The  system  Is  *ws  synchronous  and  the 
manager's  function  Is  to  locus  the  attention  ot  the  system  by  selecting 
Ihe  "best*  module  from  the  ready-fist  on  each  cycle 

We  cat  CODGER  a  whiteboard  because  II  also  implements  a 
heterarchical  system  structure,  but  (fibers  Irom  a  blackboard  In  several 
key  respects.  In  COOGER,  each  module  is  a  separate,  continuously 
running  program;  the  modulee  communlcale  by  storing  and  retrieving 
data  in  the  cental  database.  Synchronization  is  achieved  by 
primitives  In  the  data  retrieval  fadfibes  that  attorn,  for  example,  for  a 
module  to  request  data  and  suspend  execution  unfit  the  specified  data 
appears.  When  some  other  module  stores  the  desired  data,  the  ffest 
module  wtfi  be  ra- activated  and  the  data  w*  be  sent  lo  tt  With 
CODGER  a  mockjfe  programmer  thus  has  control  over  the  (low  ot 
execution  within  his  module  and  may  Implement  real-time  loops. 
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demons,  data  flows  among  cooperating  modules,  etc  CODGER  also 
nas  no  pre-avnpfied  Hsl  at  data  tesievai  specifications:  each  time  a 
module  requests  data.  H  provides  a  pattern  tor  the  data  desired  at  that 
time  We  cafi  such  a  system  a  whiteboard — it  is  heterarctmjal  Hke  a 
blackboard,  but  each  module  runs  in  paratet  with  the  module 
programmer  controttlcg  the  synchronization  and  data  retrieval 
requests  as  best  suited  lor  each  module  Like  other  recent  distributed 
Al  architectures,  whiteboards  are  stilled  to  execution  on  multiple 
processors 

5.2  Data  Storage  and  Retrieval 

Data  in  the  CODGER  database  (Local  Map)  Is  represented  In 
tokens  consisting  ol  classical  attribute-value  pairs  The  types  ol 
tokens  are  described  In  a  template  We  that  tefis  the  name  and  type  of 
each  attribute  in  tokens  of  each  type.  The  afttbutes  themselves  may 
be  the  usual  scalars  (Integers,  floating-point  values,  strings, 
enumerated  types),  arrays  (or  sets)  ol  these  types  (Including  arrays  ol 
arrays),  or  geometric  locations  as  described  below  CODGER 
automatically  maintains  certain  attributes  lor  each  token,  the  token 
type  and  id  number,  the  ‘generation  number*  as  the  token  Is  modified, 
the  time  at  which  (he  token  was  created  and  inserted  into  the 
database,  and  the  time  at  which  the  sensor  data  was  acquired  that  ted 
to  the  creation  ol  this  token.  The  LM8  Interlace  provides  facilities  lor 
boikSng  and  dissecting  tokens  and  attributes  within  a  module.  Rapid 
execution  is  supported  by  mapping  (he  module  programmer's  names 
(or  tokens  arid  attributes  onto  gtobaSy  used  Index  values  at  system 
slartup  time 

A  module  can  store  a  token  by  caring  a  subroutine  to  send  it  to  the 
LMB  Tokens  can  be  retrieved  by  constructing  a  pattern  called  a 
specification  and  calling  a  routine  lo  request  that  the  LMB  send  back 
tokens  matching  that  specification  The  specification  Is  simply  a 
boolean  expression  in  which  the  attributes  ol  each  token  may  be 
substituted;  II  a  token  s  attributes  satisfy  the  boolean  expression,  then 
the  token  ts  sent  to  the  module  that  made  the  request  For  example,  a 
module  may  specify 

to* 0ns  mlh  type  equal  to  Intersection'  and  traffic-control  equal 
to  'stop-sign' 

This  would  retrieve  a*  tokens  whose  type  and  traffic-control 
attributes  satisfy  the  above  oondfitons.  The  specification  may  Indude 
computations  such  as  mathematical  expressions,  finding  the  minimum 
value  within  an  array  attribute,  comparisons  among  attributes,  elc. 
CODGER  thus  Implements  a  general  database.  The  module 
programmer  constructs  a  specification  with  a  set  ol  subroutines  In  the 
CODGER  system. 

One  ol  the  key  features  ol  CODGER  Is  the  abfifiy  to  manipulate 
geometric  Information  One  ol  the  attribute  types  provided  by 
CODGER  Is  the  location,  which  Is  a  2  0-  or  3-0  polygon  and  a 
reference  to  a  coordinate  frame  In  which  that  polygon  Is  described 
Every  token  has  a  speckle  attribute  that  lets  the  location  of  that  object 
in  the  Local  Map.  H  appficable,  and  a  specification  can  Indude 
geometric  calculations  and  expressions.  For  example,  a  specification 
might  be: 

token*  wiki  location  w fthfn  5  unit*  al  (45.32)  (In  world  coordnmtmsj 
or 

token*  with  location  ovwrtmppng  X 

where  X  Is  a  description  ot  a  rectangle  on  the  ground  in  front  of  the 
vehicle  The  geometric  primMves  ccrrenky  provided  by  CODGER 
Include  calculation  ot  centroid,  area,  dtemeter,  convex  hud,  orientation, 
and  minimum  bounrfing  rectangle  of  a  location,  and  distance  and 
Intersection  calculations  between  a  pair  of  locations  We  believe  that 
this  kind  of  geometric  data  retrieval  capably  Is  essential  lor 
supporting  spatial  reasoning  In  mobie  robots  wfrh  muWpte  sensors. 
We  expect  geometric  spedficatens  to  be  the  most  common  type  of 
data  retrieval  request  used  In  toe  NAVLAB. 

CODGER  also  provides  lor  automate  coordinate  system 
maintenance  and  transformation  tor  these  geometric  operations  In 
the  Local  Map,  afi  coordtoatea  ol  location  attributes  are  defined 
relative  to  WORLD  or  VEFfiCLE  oootdtoafoa;  VEHICLE  coordfrtates 
are  parameterized  by  lime,  and  toe  LMB  maintains  a  time-varying 
transformation  between  WORLD  and  VEMCL£  coordtoafes 
Whenever  new  in  formation  (l.e.,  a  new  VEHKLE- to- WORLD 
transform)  becomes  avakabte.  It  is  added  to  toe  "history'  maintained  In 
the  LMB;  the  LMB  wfil  Interpolate  to  provide  Intermediate 
transformations  as  needed,  to  addMon  to  these  basic  ooordtoate 
systems,  the  LMB  Interface  atews  a  module  programmer  to  define 


local  coordinates  relative  to  <he  basic  coordinates  or  relative  to  come 
olher  local  coordinates.  Location  attributes  defined  n  a  local 
coordinate  system  are  automatically  converted  to  ibe  appropnate 
bask:  coordinate  system  when  a  token  is  stored  in  the  database 
CODGER  provides  the  module  programmer  with  a  conversion  routine 
lo  convert  any  location  to  any  spooked  ooordtoate  system 

AH  ol  the  above  tacHtttes  need  to  work  together  lo  support 
asynchronous  sensor  fusion  For  example,  suppose  we  have  a  vision 
module  A  and  a  rangefinder  module  B  whose  results  are  to  be  merged 
by  some  module  C  The  following  sequence  ol  actions  might  occur 

1  A  receives  an  image  al  time  10  and  posts  results  on  ihe 
database  al  lime  15  Although  the  calculations  were 
carried  out  in  the  camera  coordinate  system  lor  time  1 0. 
the  results  are  automatically  converted  to  the  VEHICLE 
system  at  time  10  when  the  token  Is  stored  In  the 
database 

2  Meanwhile,  8  receives  data  at  time  12  and  posl3  results 
at  time  1 7  In  a  similar  way 

3  At  time  18.  C  receives  A  s  and  B’s  results  As  described 
above,  each  such  token  will  be  tagged  wtth  the  ame  at 
which  the  sensor  data  was  gathered.  C  decides  to  use 
the  vehicle  coordinate  system  at  time  12  (B  s  time)  lor 
merging  the  data. 

4  C  requests  that  A  s  result,  which  was  stored  in  VEHICLE 
time  10  coordinates,  be  transformed  into  VEHICLE  time 
1 2  coordinates.  It  necessary,  the  LMB  will  automatically 
interpolate  coordinate  transformation  data  to  accomplish 
this.  C  can  now  merge  A's  and  B's  results  since  they 
are  in  the  same  coordinate  system  Al  Ome  23,  C  stores 
results  In  Ihe  database,  with  an  Indication  that  they  are 
stored  In  ihe  coordinate  system  ol  time  12. 

5.3  Synchronization  Primitives 
CODGER  provides  module  synchronization  through  options 
specified  lor  each  data  retrieval  request  Every  time  a  module  sends 
a  specification  to  the  LMB  to  retrieve  tokens.  It  also  specifies  options 
that  te*  how  the  LMB  should  respond  with  toe  matching  tokens: 

•  Immediate  Request  Th»  module  requests  all  tokens 
currently  in  the  database  that  match  this  speculation 
The  modike  w*  block  (l.e..  the  "request"  subroutine  in  the 
LMB  Interface  wf*  not  return  control)  until  the  LMB  has 
responded.  If  there  are  no  tokens  that  match  the 
specification,  toe  action  taken  Is  determined  by  an  option 
In  the  module's  request: 

•  Non-Blocking.  The  LMB  wHI  answer  that  there  are 
no  matching  tokens,  and  the  module  can  then 
proceed.  This  would  be  used  for  tfme-critlcrk 
modules  such  as  vehicle  control.  Example:  'Is 
there  a  slop  sign?" 

•  Blocking.  The  LMB  w*  record  this  specification  and 
compare  K  against  afi  Incoming  tokens.  When  a 
new  token  matches  Ihe  specification.  It  w*l  be  sent 
to  the  module  and  the  request  wfil  be  satis  fled 
Meanwhfie,  the  module  wfil  remain  blocked  unfit  the 
LMB  has  responded  wfth  a  token.  This  Is  Ihe  type 
ol  request  used  (or  setting  up  synchronized  sets  ol 
comnktolcaflng  modules:  each  one  wikis  lor  Ihe 
res  iris  from  the  previous  module  to  be  posted  to 
the  database.  Example:  "Wake  me  up  when  you 
see  a  stop  sign.' 

•  Standing  Request  The  module  gives  a  specification 
along  wtto  the  name  of  a  subroufine.  The  module  then 
continues  running;  the  LMB  wfil  record  toe  specification 
and  compare  It  with  afi  Incoming  tokens  Whenever  a 
token  matches,  It  wfil  be  sent  to  the  modulo.  The  LMB 
toterfaoe  wifi  Intercept  toe  token  and  execute  the 
speckled  sirirouttoa,  passing  ths  token  as  an  argument. 

This  h8S  the  effect  of  Invofdng  toe  given  subroutine 
whenever  a  token  appears  In  Ihe  database  that  matches 
toe  given  specification,  tt  can  be  used  af  system  startup 
Urns  to r  a  module  programmar  to  set  i*>  "demon’  routines 
wlthfci  toe  rrxxkke.  Example;  "Execute  that  routine 
whenever  you  see  a  stop  sign.' 


5.4  Architecture 

there  are  several  modules  that  use  the  CODGER  tools,  and  that  fit 
into  a  higher  level  architecture  The  modules  are 

•  Pilot  Looks  at  the  map  and  at  cunei't  vehicle  position  to 
predict  road  location  lor  Vision  Plans  paths. 

•  Color  Vision  Warts  tor  3  prediction  from  the  Pilot,  waits 
until  the  vehicle  is  in  the  best  position  to  take  an  image  ol 
that  section  of  the  road,  returns  road  iocation 

•  Obstacle  Avoidance  Gets  a  request  trom  the  Pilot  to 
;heck  a  part  jl  the  road  tor  obstacles.  Returns  a  list  o( 
obstacles  on  or  near  lhat  chunk  ol  the  road. 

•  Holm  Gets  planned  path  Irom  Pilot,  converts  polyline 
path  into  smooth  arcs,  steers  vehicle 

•  Graphics  and  Monitor  Draws  or  pdnts  position  of  vehicle, 
obstacles,  predicted  and  perceived  road. 

There  are  three  other  modules  in  our  architecture  but  not  yet 
implemented 

•  Captain  Talks  to  the  user  and  provides  high-level  route 
and  mission  constraints  such  es  "avoid  area  A"  or  "go  by 
road  B" 

•  Map  Navigator  Maintains  a  map,  does  global  path 
planning,  provides  long-term  direction  to  the  Pitot 

•  Lookout  Looks  lor  landmarks  and  objects  of  importance 
to  the  mission 

These  modules  use  CODGER  to  pass  information  about  "driving 
units'  A  driving  unit  Is  a  short  chunk  of  the  road  or  terrain  (in  our 
case  4  meters  long)  treated  as  a  unit  tor  perception  and  path  planning. 
The  Pilot  gives  driving  unit  predictions  to  Color  Vision,  which  returns 
an  updated  driving  unit  location.  Obstacle  Detection  then  sweeps  a 
driving  unit  for  oosiacies.  The  PHol  takes  the  driving  unit  and 
obstacles,  plans  a  path,  and  hands  the  path  off  to  the  Helm  The 
whole  process  Is  set  up  as  a  pipeline,  in  which  Color  Vision  is  looking 
ahead  3  driving  units.  Obstacle  Detection  Is  looking  2  driving  units 
ahead,  and  path  planning  at  the  next  unit  It  for  any  reason  some 
stage  slows  down,  all  following  stages  of  the  pipeline  have  to  watt. 
So.  for  instance,  if  Color  Vision  Is  waiting  for  the  vehicle  to  come 
around  a  bend  so  It  can  see  down  the  road.  Obstacle  Detection  witt 
finish  its  current  unit  and  will  then  have  to  wait  for  Color  Vision  to 
proceed  In  an  extreme  case,  the  vehicle  may  have  to  come  to  a  haft 
until  everything  clears  up  All  planned  paths  Include  a  deceleration  to 
a  stop  at  the  end.  so  if  no  new  path  comes  along  to  overwrite  the 
current  path  the  vehicle  wilt  stop  before  driving  into  an  area  that  has 
not  been  seen  or  cleared  of  obstacles. 

In  our  current  system  and  test  area,  three  driving  units  Is  too  far 
ahead  for  Color  Vision  to  look,  so  both  Color  Vision  and  Obstacle 
Detection  are  looking  at  the  same  driving  unit.  Obstacle  Detection 
looks  at  an  area  enough  larger  than  the  Pilot's  predicted  driving  unit 
location  that  the  actual  road  Is  guaranteed  to  be  covered.  Another 
practical  modification  Is  to  have  Obstacle  Detection  look  at  the  closest 
driving  unit  also,  so  if  a  person  walks  onto  the  road  Immediately  In 
front  of  the  vehicle  he  will  be  noticed.  Our  system  wll  try  to  plan  a 
path  around  obstacles  while  remaining  on  the  road.  II  that  Is  not 
possible,  it  wilt  come  to  a  halt  and  wait  for  the  obstacle  to  move  before 
continuing. 

6.  Conclusions  and  Future  Work 

The  system  described  here  works.  Since  quashing  the  last  ol  our 
iknown)  bugs,  it  has  successfully  driven  the  Navtab  many  tens  of 
limes,  processing  thousands  of  color  and  range  Images  without 
running  off  the  road  or  hitting  any  obstacles.  CODGER  has  proved  to 
be  a  useful  toot,  handling  a  lot  of  the  detaHs  of  communications  and 
geometry  Module  developers  have  been  able  to  build  and  lest  their 
routines  in  isolation,  wtth  lelattvefy  Utile  Integration  overhead.  Yet 
there  are  several  areas  that  need  much  more  work. 

Speed.  We  drive  the  Navtab  at  10  cnvsec,  a  slow  shuffle.  Part  ot 
the  roason  tor  this  Is  because  our  test  road  is  narrow  and  winding,  and 
part  ot  the  reason  Is  that  we  deliberately  concentrate  on  competence 
rather  than  speed  But  taster  motion  Is  always  more  interesting,  so  we 
are  pursuing  several  ways  ot  increasing  speed.  One  bottleneck  is  the 
computing  hardware  We  are  mounting  a  Warp,  CMU's  experimental 


high  speed  processor,  on  the  Navtab  The  Warp  wit  give  us  a  factor 
ol  100  more  processing  power  than  a  Sun  for  color  and  range  image 
processing  At  the  same  time,  we  are  looking  at  improvements  to  the 
software  architecture  We  need  a  more  sophisticated  path  planner, 
and  we  need  to  process  Tnages  that  are  more  closely  spaced  than  the 
length  ot  a  driving  unit.  Also,  as  the  vehicle  moves  more  quickly,  our 
simplifying  assumption  that  steering  is  instantaneous  and  that  the 
vehicle  moves  along  circular  arcs  becomes  more  sedously  flawed 
We  are  looking  at  other  kinds  ot  smooth  arcs,  such  as  clothofds. 

Map.  One  particular  reason  tor  the  slow  speed  is  that  the  Pilot 
assumes  straight  roads  We  need  to  have  a  description  that  allows  tor 
curved  roads,  with  some  constraints  on  maximum  curvature  The  next 
steps  will  include  building  maps  as  we  go.  so  that  subsequent  runs 
over  the  same  course  can  be  faster  and  easier 

Cross  country  travel.  Travel  on  roads  Is  only  halt  the  challenge 
Die  Navlab  should  be  able  to  leave  roads  and  venture  cross  country 
Our  plans  call  lor  a  fully  integrated  on-road/oti-road  capability 

Intersections.  Current  vision  routines  have  a  built  in  assumption 
that  there  is  one  road  in  the  scene  When  the  Navtab  comes  to  a  tork 
in  the  road,  vision  wilt  report  one  or  the  other  ot  the  forks  as  the  true 
road  depending  on  which  looks  bigger  It  will  be  important  to  extend 
the  vision  geometry  to  handle  intersections  as  well  as  straight  roads 
We  already  have  this  ability  on  our  sidewalk  system,  and  will  bring  that 
over  to  the  Navlab. 

Landmarks.  Especially  as  we  venture  olt  roads,  it  wf*  become 
increasingly  important  to  be  able  to  update  our  position  based  on 
sighting  landmarks.  This  Involves  map  and  perception  enhancements, 
plus  understanding  how  to  share  limited  resources,  such  as  the 
camera,  between  path  finding  and  landmark  searches. 

Software  Development  Our  current  blackboard  system  can 
manipulate  primitive  data  elements  but  has  no  concept  ol  data 
structures  made  up  of  tokens  on  the  blackboard  We  need  aggregate 
data  types  tor  representing  complex  3D  geometric  descriptions  of 
ob|ects  tor  recognition  We  wfll  also  be  Implementing  a  LISP  interlace 
to  our  blackboard  so  that  not  ail  modules  need  to  be  written  In  C 

Integration  with  Work  trom  Other  Sites.  There  are  other 
universities  and  research  groups  cooperating  wtth  Carnegie  Mellon 
through  DARPA's  Strategic  Computing  Vision  program.  We  plan  to 
incorporate  some  of  their  programs  Into  the  Navtab  system  in  the 
coming  years  as  it  evolves  Into  the  "New  Generation  Vision  System" 
which  is  that  goal  ol  that  program. 
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Tne  current  upper  i  im i t  on  the 
capacity  >f  computing  systems  is 
determined  by  our  ability  to  design 
computers,  not  by  demand.  Advances  in 
circuit  fabrication  technology  have 
recently  made  possiole  the  construction 
of  a  new  type  of  computing  engine  called 
the  Connection  Machine  which  contains 
tens  of  thousands  of  tiny  processors. 
Each  processor  is  capaDle  of  computing 
concurrently,  so  the  machine  is  much 
faster  than  a  conventional  computer. 

3ackq  round 

The  first  prototype  of  the  Connection 
system,  with  65,536  processors, 
was  constructed  by  Thinking  Machines 
Corporation  under  contract  from  DARPA  in 
1985.  Even  this  first  prototype,  the 
CM  1 ,  is  one  of  the  world's  fastest 
computers,  and  it  is  currently  being 
applied  to  problems  in  fluid  dynamics, 
database  retrieval,  design  automation, 
image  processing,  simulation,  and 
artificial  intelligence.  A  commercial 
version  of  the  CM1  was  introduced  in 
spring  of  1986,  and  is  currently  being 
sold  and  manufactured  by  Thinking 
Machines  Corporation. 

The  key  idea  of  the  Connection  Machine  is 
to  combine  memory  and  processing  into  a 
single  cell  which  can  be  replicated  many 
times.  Most  computers  today  are  based  on 
a  two-part  design  where  the  memory  and 
processing  functions  are  performed  by 
separate  units.  This  two-part  design 
made  a  great  deal  of  sense  in  the  days 
when  processing  was  realized  by 
relatively  expensive  and  fast  vacuum 
tubes  and  memory  was  realized  by 
relatively  slow  and  inexpensive  delay 
lines.  Since  the  two  functions  of  the 
computer  were  implemented  by  essentially 
different  technologies,  they  were 
implemented  separately.  The  system  was 
programmed  in  such  a  way  as  to  keep  the 
expensive  vacuum  tubes  relatively  busy. 
Programmers  were  trained  to  break  up  a 
problem  into  a  series  of  sequential 
operations,  so  that  the  data  could  be 
processed  one  word  at  a  time.  For 
example,  to  simulate  that  flow  of  a  fluid 
on  a  conventional  machine  the  volume  of 
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the  velocities  of  various  points  in  space 
would  be  stored  in  the  memory  and  cut  up 
into  cells.  To  simulate  the  change  in 
fluid  over  a  moment  of  time,  the 
programmer  must  arrange  to  sequentially 
shift  the  cells  to  the  processing  unit 
one  at  a  time  for  computation.  Although 
the  computation  would  be  modelling  a 
system  that  is  f undamen t a  1 1 y  parallel, 
namely  the  flow  of  the  fluid,  the 
programmer  would  realize  this  by  a  series 
of  sequential  operations. 

This  sequential  method  of  solving 
problems  made  a  great  deal  of  sense  m 
the  1940's,  since  it  allowed  only  one 
processing  unit  for  a  large  amount  of 
memory.  The  economics  of  the  technology 
at  the  time  were  such  that  the  single 
processing  unit  was  as  large  and 
expensive  as  many  thousands  of  words  of 
memory.  Today  we  still  use 

word-at-a-time  processing  in  almost  all 
our  computers  even  though  the  technical 
reasons  for  using  it  have  largely 
vanished.  Both  memory  and  processing 
elements  today  are  built  of  the  same 
fundamental  material;  usually  silicon. 
They  can  be  fabricated  on  the  same 
integrated  circuit.  In  spite  of  this,  a 
modern  single  chip  computer  is 
essentially  just  a  scaled-down  version  of 
the  roomful  of  vacuum  tubes  and  mercury 
delay  lines  that  represented  a  computer 
in  the  1950's.  The  organization  of 
programs  in  terms  of  streaming  memory 
past  a  single  processor,  although 
originally  developed  for  reasons  of 
efficiency,  has  become  a  serious 
performance  handicap.  Today  most  of  the 
silicon  in  a  computer  is  in  the  memory, 
and  most  of  this  is  idle. 

One  way  to  speed  up  a  computer  is  to 
increase  the  speed  of  the  primitive 
operations.  This  is  difficult  to  do 
beyond  a  certain  point.  The  fastest 
conventional  machines  already  have 
operation  times  comparable  to  the  time 
required  for  light  to  travel  from  one 
side  of  the  processor  to  the  other. 
While  some  improvement  can  be  made  by 
making  the  processing  components 
physically  smaller,  the  operation  times 
of  processors  are  getting  close  to  their 
practical  limits.  It  is  becoming 

increasingly  difficult  to  improve  the 
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•?rt  jcmance  of  computers  without 
.  ■'.  d  imen  t  a  1  L  y  .'hanging  the  design.  Since 
.  ;  t  speed  improvements  require  the  use 

"  faster  components  ari'i  lenser 
i.'xajmj,  i  u  ; e  computers  today  are 
.  ,  i  :  f  c:  a  n  t y  less  economical  than  their 
"  a  .  L  •  >-  scale  counterparts  in  terms  of 


parallel.  We  ;a.i  "axe  advantage  of  the 

•c'd^i's  are  fabricated  on  the  same 
s.ucon  chip  and  produce  a  simple 
... r  ■  lessor  memory  unit  that  can  be 
replicated  tens  of  thousands  of  times. 
W  can  also  take  advantage  of  a  more 
n.ituraL  mapping  of  the  problem  into  the 
“'a  chine  by  e  1  im  mat  i  ng  the  need  to 
arrange  ".he  computations  in  serial  order. 

l.ne  construction  of  a  machine  with 
-.''•.reds  of  oaousands  of  processors  is 
made  possible  by  the  availability  of  very 
larie  scale  integrated  circuits  with 
.oirods  of  thousands  of  transistors  on  a 
c.ngle  chip.  But  gust  naving  many 

;  r->  lessors  is  not  sufficient.  Very  few 
;  '•■rrast  ing  applications  decompose  into 
-  'tally  independent  subproblems;  most 
recuire  communication  between  the 
rices  sing  elements.  For  example,  in  a 
"tree-dimensional  fluid  flow  problem  each 
."it  if  volume  must  communicate  with  its 
immediate  neighbors  in  three-dimensional 
space.  In  other  problems,  such  as  a 
.arge  lata  base  proDlem,  the  pattern  of 
the  communication  can  be  much  more 
■implicated,  depending  entirely  upon  the 
.intent  if  the  data.  In  fact,  many 
pribiems  require  dynamically  changing 
patterns  of  communication  that  rearrange 
hiring  the  course  of  computation. 

I'nis  lei  is  to  the  second  important 
element  of  the  connection  Machine 
architecture:  the  connections.  Each 

processor i memory  cell  is  linked  by  a 
;  irnnun.  i  c  a  t  i  ons  network  to  all  other  cells 
in  tne  macnine.  This  arrangement  is 
similar  to  giving  each  cell  a  telephone 
witn  whim  it  can  call  up  any  other  cell 
witn  wm ch  it  needs  to  communicate.  The 
packet  switch  communication  network 
serves  the  role  of  the  telephone  system, 
.'ells  can  communicate  whenever  they  know 
me  another's  "telephone  numbers."  In  a 
"..tree-dimensional  problem,  such  as 
modelling  the  flow  of  fluid  over  an 
airplane  wing,  each  cell  is  given  the 
numbers  if  the  cell  representing  its 
t  i  ree-d  imens lona 1  neighbors.  Cells  do 
not  have  a  physical  three-dimensional 
pattern  of  wiring,  but  many  applications 
involve  a  more  complex  pattern  of 
■connections.  For  an  example  of  an 

application  that  requires  a  more  complex 
pattern  of  communications,  consider  the 
simulation  of  an  electronic  circuit  with 
a  Hundred  thousand  transistors.  Such 
simulations  are  common  and  commercially 
important  during  t-he  design  of  integrated 


circuits.  In  such  an.  applicati 
natural  to  use  one  processing  mem 
to  represent  the  state  of  sap 
hundred  thousand  transistors. 
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natural  pattern  of  commun i c j t  i  on  iepends 
on  the  exact  wiring  pattern  of  the 
transistors.  If  two  processor  cells 
simulate  two  transistors  which  are 
connected  in  the  diagram,  then  the 
processor  cells  must  communicate.  when 
the  voltage  level  on  one  transistor 
changes,  this  change  is  propagated  to  the 
other  transistor  through  the  connection. 
In  such  an  application  the  processing 
cells  are  assigned  to  transistors 
arbitrarily,  and  each  processing  cell  is 
given  the  telephone  numbers  of  the 
transistors  to  which  it  is  connected. 
Thus  the  communications  structure  of  the 
processing  elements  exactly  matches  the 
natural  structure  of  the  problem. 

In  more  complex  algorithms,  such  as 
those  that  occur  in  the  study  of 
artificial  Intelligence,  the  connection 
pattern  may  actually  change  during  the 
course  of  a  computation.  For  example, 
each  processor  may  represent  a  simple 
concept,  with  the  connections  between  the 
processors  representing  the  relat  lonsnips 
between  the  concepts.  One  processor  may 
represent  the  concept  "Socrates,"  another 
"man,"  and  another  "mortal."  A 
connection  between  the  processor 
representing  "Socrates*  and  the  processor 
representing  "man"  represents  the  concept 
that  "Socrates  is  a  man."  And, 
similarly,  a  connection  between  man  and 
mortal  may  represent  the  statement  that 
"all  men  are  mortal."  Such  a  machine 
would  be  able  to  deduce  the  concept  that 
Socrates  was  mortal  by  following  the 
chain  of  connections.  The  example  is 
simple  and  contrived.  But  in  a  real 
artificial  intelligence  application,  a 
simple  common-sense  knowledge  base  might 
consist  of  hundreds  of  thousands  of  such 
assertions;  for  example,  assertions  like 
"dogs  are  animals"  and  "apples  are  often 
red."  From  such  a  knowlege  base  a 
program  may  be  required  to  make  hundreds 
of  common-sense  inferences  per  second. 
Each  time  a  new  fact  is  learned,  a  new 
connection  must  be  formed. 

The  implementation  of  the 
communication  network  is  the  most 
difficult  technical  problem  in 
constructing  a  Connection  Machine  system. 
For  example,  in  the  first  65,536  of  the 
machine,  new  connections  are  formed  at  a 
rate  of  about  one  hundred  million  each 
second.  This  is  comparable  to  the 
switching  capacity  of  the  entire 
telephone  exchange  for  the  city  of  New 
York  . 

The  key  component  that  makes  this  all 
work  is  an  integrated  circuit  that 
integrates  16  simple  processing  units  and 
a  communications  router.  Each  of  the 
processinc  units  is  a  very  simple 
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computer  with  4,000  aits  of  memory.  The 
difficult  part  is  the  router.  There  are 
4,096  chips  m  the  65,536  processor 
system.  Packaging  considerations  make  it 
impractical  to  have  a  direct  connection 
Between  all  pairs  of  chips.  Instead, 
each  router  is  connected  directly  to  only 
12  others  in  the  system.  Two  processors 
within  a  single  chip  may  communicate 
directly,  out  processors  on  different 
Chips  may  need  to  communicate  through 
intermediaries.  This  is  the  function  of 
a  router.  It  is  the  responsibility  of  a 
router  to  forward  an  incoming  message 
from  one  chLp  to  another.  The  pattern  of 
wiring  is  such  that  six  such  forwarding 
steps,  on  the  average,  are  required  to 
communicate  between  two  chips.  Even  with 
all  the  forwarding  steps,  the 
transmission  of  a  message  from  one 
processor  to  another  requires  less  than 
100  microseconds.  Each  packet  is 
transmitted  as  a  stream  of  bits  that  are 
forwarded  through  the  system  m  such  a 
way  that  the  beginning  of  the  stream  is 
usually  delivered  before  the  end  is  e  en 
transmitted. 

The  entire  65  ,  536  processor  CM  1 , 
including  processor/memory  units  and  the 
communication  networks,  fits  into  a  cube 
approximately  five  feet  on  a  side.  It  is 
■controlled  by  a  conventional  host 
computer  so  that  the  operating  system  and 
user  interface  seen  by  t ne  user  are  those 
jf  the  host.  In  fact,  the  Connection 
Machine  connects  to  the  host  computer  in 
much  the  same  way  as  a  conventional 
memory  unit.  The  processor/memory  cells 
can  be  accessed  simply  as  memory  cells  by 
the  host.  This  arrangement  allows  simple 
integration  of  parallel  computing  and 
existing  software,  since  data  structures 
may  be  shared  by  both  the  Connection 
Machine  and  the  conventional  von  Neumann 
host. 

The  design  of  the  65,536  processor 
machine  was  optimized  for  simplicity  and 
reliability  rather  than  speed.  It  runs 
at  a  relatively  slow  clock  rate  of  250 
nanoseconds,  as  compared  to,  say,  25 
nanoseconds  for  a  typical  supercomputer. 
There  are  no  exotic  technologies  to  the 
machine.  The  custom  chip  is  built  with  a 
simple  integrated  circuit  process  similar 
to  that  used  in  personal  computers  and 
pocket  calculators.  The  circuit  board's 
cooling,  packaging,  and  power  systems  are 
all  similar  to  those  found  in 
conventional  computers.  Even  with  this 
conservaive  technology,  the  Connection 
Machine  is  able  to  achieve  computing 
rates  of  greater  than  1000  million 
instructions  per  second,  more  than  100 
times  the  speed  of  a  typical  large 
computer.  The  instruction  set  of  the 
machine  is  optimized  for  operating  on 
problems  that  involve  images  and  words 
rather  than  just  numbers.  For  this  type 
of  problem  the  machine  is  able  to  achieve 
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abstract 

The  paper  summarizes  the  current  researcn  in  the  Computer 
Vision  Research  Laboratory  at  the  University  of  Michigan. 
The  laboratory  concentrates  on  developing  generic  vL  on 
algorithms  for  industrial  applications.  Generic  vision  algo¬ 
rithms  cam  be  applied  to  a  wide  variety  of  inspection  prob- 
iems.  The  paper  includes  a  discussion  of  the  current  state 
of  the  machine  vision  industry  and  provides  recommenda- 
tioas  for  improving  the  transfer  of  vision  technology  from 
research  to  practice. 


1  Introduction 

The  University  of  Michigan  recently  formed  a  laboratory, 
called  the  Computer  Vision  Research  Laboratory,  within  the 
Department  of  Electrical  Engineering  and  Computer  Sci¬ 
ence  for  research  in  computer  vision.  The  Computer  Vision 
Research  Laboratory  will  evolve  through  interaction  among 
the  researchers  of  various  related  disciplines  and  through 
inductrial  and  government  contacts. 

The  research  program  in  the  Computer  Vision  Labora¬ 
tory  concentrates  on  the  development  of  generic  vision  algo¬ 
rithms  that  can  be  applied  to  a  variety  of  situations.  Generic 
perception  algorithms  are  simple,  reliable  modules  that  hide 
the  details  of  perception  in  a  packaged  hardware  or  software 
component  that  can  be  used  by  applications  specialists  who 
are  not  themselves  expert  vision  researchers.  Generic  vi¬ 
sion  algorithms  are  not  developed  for  specific  applications, 
but  are  designed  to  solve  a  vision  task  that  is  part  of  many 
different  inspection  problems  that  occur  in  different  appli 
cations  in  different  industries.  In  developing  generic  vision 
algorithms,  researchers  leverage  their  efforts  by  providing 
solutions  to  classes  of  vision  tasks  while  removing  the  details 
of  the  application  from  the  research  efforts.  This  insulates 
the  vision  researchers  from  the  details  of  the  application 
and  allows  them  to  concentrate  on  understanding  the  vision 
problems. 

Ano'  her  aspect  of  research  on  generic  vision  algorithms 
is  that  the  algorithms  are  classified  according  to  properties 
that  are  meaningful  in  the  context  of  the  emerging  knowl¬ 
edge  of  vision  fundamentals,  as  opposed  to  classification 
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according  to  applications  areas.  There  are  no  automotive 
vision  algorithms  or  aerospace  vision  algorithms;  vision  al¬ 
gorithms  are  not  specific  to  any  one  industry.  Vision  algo¬ 
rithms  are  defined  by  the  p'operties  of  the  vision  processing 
task;  that  is,  in  terms  that  depend  on  the  vision  processing 
itself,  rather  than  on  some  intended  use  for  th°  algorithm. 


The  nature  of  the  problems  of  applying  vision  to  spe¬ 
cific  applications  has  not  been  widely  perceived  The  cur¬ 
rent  macroeconomic  system  of  machine  vision  is  ^appro¬ 
priate.  The  solution  is  to  place  machine  vision  technology 
in  the  hands  of  the  end  users  who  are  f?  miliar  with  the  re¬ 
quirements  of  their  environment.  Vision  technology  should 
be  like  any  other  instrumentation:  plug  it  in  and  use  it  to 
make  measurements.  It  is  not  practical  for  the  developers  of 
machin'*  vision  applications  to  perform  the  development  spe¬ 
cific  to  the  application  and  to  develop  the  “instrumentation" 
required  for  the  development  at  the  same  time.  To  change 
the  current,  untenable  system,  vision  researchers  must  pro¬ 
duce  generic  vision  algorithms  that  can  be  used  by  vision 
systems  developers  in  a  variety  of  applications. 

Researchers  are  currently  addressing  problems  related 
to  vision  engineering  but  without  much,  if  any,  concern  for 
designing  machine  vision  systems.  The  focus  of  the  Com¬ 
puter  Vision  Research  Laboratory  on  vision  algorithms  that 
are  generic,  rather  than  applications  specific,  meets  needs 
not  addressed  by  other  research  institutions.  The  Com¬ 
puter  Vision  Research  Laboratory  provides  a  focus  for  the 
synergistic  interaction  of  researchers  aided  by  interactions 
with  industry.  The  goal  of  research  in  the  laboratory  is  to 
address  fundamental  problem i  blocking  advances  in  the  de¬ 
sign  of  machine  vision  systems  for  various  applications.  The 
research  addresses  basic  issuzs  in  computational  vision  re¬ 
search  and  the  design  of  systems  that  can  perform  vision 
tasks  in  real  time  for  a  given  application.  New  engineering 
techniques  for  designing  systems  are  being  developed  that 
can  perform  tasks  in  an  unstructured  environment.  The 
research  may  also  advance  the  understanding  of  biological 
vision  systems. 


2  Research  Problems 


Visual  perception  is  the  process  of  interpreting  measure¬ 
ments  such  as  light  intensity  and  range  that  relate  to  the 
projection  of  surfaces  in  a  scene  onto  the  image  plane.  The 
central  problem  in  vision  is  the  reconstruction  of  surface 
structure  and  properties  from  the  projection  onto  the  image 
plane.  This  paper  outlines  some  of  the  research  problems 
currently  being  explored  in  the  Computer  Vision  Research 
Laboratory. 

•  Dynamic  vision 

•  Range  images 

•  Knowledge-based  perception  systems 

•  Computer  architectures  for  vision  algorithms 

•  Sensor  integration 

•  Bridging  the  gap  between  research  results  and  appli¬ 
cations 

3  Dynamic  Vision 

Motion  provides  valuable  clues  to  surface  structure.  Al¬ 
gorithms  for  interpreting  sequences  of  images  can  provide 
measurements  that  are  very  useful  for  applications.  Image 
flow  research  has  many  applications  including  passive  sens¬ 
ing  systems  for  autonomous  mobility,  machine  inspection  of 
surface  structure,  passive  sensors  for  aircraft  and  satellite 
Jocking  systems,  and  image  compression. 

3.1  Background 

The  last  few  years  have  seen  increasing  interest  in  dynamic 
scene  analysis.  The  input  to  a  dynamic  scene  analysis  sys¬ 
tem  is  a  sequence  of  images.  A  major  problem  in  a  computer 
vision  system  is  to  recover  the  information  about  objects  in 
a  scene  from  images.  This  problem  cannot  be  solved  without 
some  assumptions  about  the  world.  A  sequence  of  frames 
provides  the  additional  dimension  of  time  for  recovering  the 
information  about  a  3-dimensional  world  that  is  lost  in  the 
projection  process.  Multiple  views  of  a  moving  object  ac¬ 
quired  using  a  stationary  camera  may  allow  recovery  of  the 
structure  of  the  object  [45).  A  mobile  camera  may  be  used 
to  acquire  information  about  the  structure  of  the  stationary 
objects  in  a  scene  using  optical  flow  jilj,  auual  motion  svereo 
[25,321,  and  other  methods  (21). 

Many  researchers  in  the  psychology  of  vision  support  the 
recovery  of  information  from  image  sequences,  rather  than 
from  a  single  image  [11,31).  Gibson  (11|  argued  in  support 
of  active  information  pick-up  by  the  observer  in  an  environ¬ 
ment.  Neisser  [31 )  proposed  a  model  in  which  the  perceptual 
processes  continually  interact  with  the  incoming  informa¬ 
tion  to  verify  expectations  formed  on  the  basis  of  available 
information.  In  computer  vision  systems,  the  power  of  ex¬ 
ploiting  motion,  even  with  such  noise  sensitive  approaches 
as  difference  pictures  and  accumulated  difference  pictures, 
has  been  demonstrated  on  complex,  real  world  scenes  [20j. 
Many  researchers  are  addressing  the  problem  of  recovering 


information  in  dynamic  scenes;  but  due  to  the  legacy  of 
static  scenes,  most  researchers  are  approaching  the  recovery 
problem  using  just  two  or  three  frames  of  a  sequence.  This 
restricts  the  results  to  quasi-dynamic  scene  analysis,  rather 
than  true  dynamic  scene  analysis.  The  information  recovery 
process  requires  constraints  about  the  scene.  The  analysis 
based  on  small  numbers  of  frames  rests  on  assumptions  that 
ignore  the  most  important  information  in  dynamic  scenes. 

The  viewpoint  of  the  Computer  Vision  Research  Lab¬ 
oratory  is  that  image  understanding  is  a  dynamic  process. 
Dynamic  vision  algorithms  cope  with  the  error-filled  visual 
world  by  exploiting  redundant  information  in  the  image  se¬ 
quence.  Current  research  efforts  are  developing  a  qualitative 
approach  to  vision  that  uses  only  relative  information  avail¬ 
able  in  a  sequence  to  infer  relationships  between  objects  in 
a  scene. 

3.2  Segmentation 

In  many  dynamic  scene  analysis  systems,  the  goal  is  to  rec¬ 
ognize  moving  objects  and  to  find  their  motion  characteris¬ 
tics.  If  the  scene  is  acquired  using  a  stationary  camera,  then 
segmentation  generally  refers  to  the  separation  of  moving 
components  from  stationary  components  in  the  scene  and 
identification  of  individual  moving  objects  based  on  veloc¬ 
ity  or  some  other  characteristic.  For  the  case  of  a  moving 
camera,  the  segmentation  task  may  be  the  same  as  above 
or  may  involve  further  segmentation  of  the  stationary  com¬ 
ponents  of  the  scene  by  exploiting  the  motion  of  the  cam¬ 
era.  Most  research  efforts  for  the  segmentation  of  dynamic 
scenes  have  been  concerned  with  the  extraction  of  the  im¬ 
ages  of  the  moving  objects  observed  by  a  stationary  camera. 
It  has  long  been  argued  by  researchers  in  perception  [11,45] 
that  motion  cues  aid  the  segmentation  process.  Computer 
vision  techniques  for  segmenting  dynamic  scenes  perform 
well  in  comparison  to  those  for  the  segmentation  of  station¬ 
ary  scenes.  The  segmentation  of  moving  camera  scenes  into 
their  stationary  and  nonstationary  components  has  received 
attention  only  recently  (22).  The  major  problem  in  the  seg¬ 
mentation  of  moving  observer  scenes  is  that  every  surface 
in  the  scene  shows  motion  in  the  image.  For  separation 
of  moving  object  images,  the  motion  component  assigned 
to  various  surfaces  in  the  images  due  to  the  motion  of  the 
camera  must  be  removed.  The  fact  that  the  image  motion 
of  a  surface  depends  on  its  distance  from  the  camera  and 
the  surface  structure  complicates  the  situation. 

Jain  developed  techniques  for  the  segmentation  of  dy¬ 
namic  scenes  [19,20,22].  These  techniques  have  been  used 
in  many  applications.  Sandia  Laboratories  is  developing 
systems  for  tracking  objects  for  Army  applications  using 
techniques  based  on  differencing  using  a  likelihood  ratio. 
Recently  research  led  to  a  new  approach  for  segmentation 
using  accumulative  difference  pictures  that  may  be  imple¬ 
mented  easily  on  sper ' -1  hardware  [23]. 

3.3  Image  Flow 

Image  flow  is  the  velocity  field  in  the  image  plane  that  arises 
due  to  the  projection  of  moving  patterns  in  the  scene  onto 
the  image  plane.  The  motion  of  patterns  in  the  image  plane 


may  be  due  to  the  motion  of  the  observer,  the  motion  of 
objects  in  the  scene,  or  both.  The  motion  may  also  be  ap¬ 
parent  motion  where  a  change  in  the  image  between  frames 
gives  the  illusion  of  motion  [36). 

The  intent  of  this  research  is  to  discover  new  models 
for  image  flow  that  will  yield  new  algorithms  for  image  flow 
estimation  and  analysis.  Constraint  equations  that  better 
model  the  interactions  between  changes  in  object  position 
and  the  illumination  and  surface  reflectance  characteristics 
will  naturally  result  in  better  algorithms  and  better  under¬ 
standing  of  existing  algorithms. 

Prior  image  flow  research  developed  an  image  flow  equa¬ 
tion  for  smooth  patterns  of  image  irradiance  and  smooth  ve¬ 
locity  fields  27).  The  equation  was  extended  first  to  image 
irradiance  patterns  with  discontinuities  and  then  to  veloc¬ 
ity  fields  with  discontinuities  [55,56,60).  Future  research  in 
image  flow  constraint  equations  will  aim  to  increase  our  un¬ 
derstanding  of  image  flow  characteristics.  This  will  lead  to 
new  algorithms  for  image  flow  estimation  that  incorporate 
the  new  continuity  equations.  Restrictions  on  the  situations 
in  which  existing  continuity  equations  can  be  used  will  be 
discovered  and  these  insights  will  improve  the  performance 
of  existing  image  flow  estimation  algorithms  by  pin  pointing 
the  situations  where  the  algorithms  cannot  be  used. 

3.4  Motion  Stereo 

Depth  determination  is  a  continuing  problem  in  computer 
vision.  Research  in  this  area  is  motivated  by  the  need  for 
target  tracking,  autonomous  vehicles  control,  visual  pros- 
theses  for  the  blind,  realistic  flight  trainers,  and  models  of 
the  human  visual  system.  There  is  a  plethora  of  dep...  de¬ 
termination  techniques.  Many  different  stereo  systems  for 
depth  determination  have  been  developed  just  in  the  last 
few  years. 

Stereo  information  can  also  be  obtained  using  a  single 
moving  camera.  Jain  and  O’Brien  [25,32|  map  images  into  a 
complex  log  space  where  the  movement  of  the  objects  in  two 
dimensions  due  to  the  camera  motion  becomes  a  translation 
along  one  axis  in  the  new  space.  Given  this  phenomenon, 
the  motion  correspondence  problem  is  greatly  reduced,  since 
only  a  small  strip  of  the  new  space  needs  to  be  searched 
[18).  This  constraint  is  similar  to  the  epipolar  constraint 
in  stereo.  In  addition,  the  transform  is  scale  and  rotation 
invariant.  It  also  has  an  analog  in  the  human  visual  system: 
the  mapping  of  the  retinal  space  into  the  striate  cortex  is 
very  closely  approximated  by  the  CLM.  Figure  1  shows  an 
image  and  its  mapping. 

Optical  flow  has  been  studied  with  the  aim  of  recover¬ 
ing  information  about  the  environment  and  the  motion  of 
the  observer.  The  egomotion  complex  logarithmic  mapping 
(ECLM)  exploits  some  characteristics  of  optical  flow  with¬ 
out  computing  it.  The  mapping  combines  scale,  projection, 
and  rotation  invariances  of  complex  log  mapping  with  the 
characteristics  of  optical  flow.  This  mapping  is  useful  in  seg¬ 
mentation  of  image  sequences  to  recover  images  of  moving 
objects.  The  distance  to  stationary  objects  can  also  be  com¬ 
puted  from  the  egomotion  complex  logarithmic  mapping. 
Using  this  mapping,  motion  stereo  and  segmentation  car 
be  achieved  in  one  step.  The  feasibility  of  the  approach  has 


Figure  1:  An  image  and  its  complex  log  mapping  are  shown 
in  this  figure.  The  mapping  is  similar  to  the  retino-striate 
mapping  in  primate  visual  systems.  This  mapping  is  very 
useful  in  segmentation  and  depth  recovery  in  dynamic  vi¬ 
sion. 


been  demonstated.  The  next  step  is  to  study  its  application 
in  dynamic  scenes.  This  research  should  yield  important 
results  for  the  navigation  of  mobile  robots. 

3.5  Motion  Trajectories 

Iterative  algorithms  for  determining  trajectories  of  points  in 
an  extended  frame  sequence  using  path  coherence  are  being 
developed.  The  emphasis  in  this  approach  is  to  exploit  mo¬ 
tion  characteristics  for  establishing  correspondence  without 
assuming  rigidity  of  objects.  A  greedy  exchange  algorithm 
has  been  developed  to  implement  this  idea  [41],  The  results 
of  applying  this  algorithm  to  a  sequence  from  the  Superman 
movie  are  shown  in  Fig.  2. 


Figure  3:  This  figure  shows  results  of  our  road  edge  detec¬ 
tion  algorithm.  For  the  road  shown  in  the  top  ft  corner, 
the  road  edges  are  shown  in  the  right  bottom  corner. 


Figure  2:  Three  frames  from  the  Superman  sequence  are 
shown  in  this  figure.  Points  on  the  head  and  belt  of  the  three 
soldiers  running  towards  the  camera  were  tracked  using  a 
greedy  exchange  algorithm.  The  trajectories  are  also  shown 
here. 


Another  approach  to  finding  trajectories  that  will  deter¬ 
mine  motion  events  is  being  studied.  In  this  approach,  mo¬ 
tion  parameters  are  recovered  using  constraints  imposed  by 
the  equations  for  the  motion  of  points  [15|.  The  algorithm 
uses  successive  refinement  to  determine  the  trajectories  of 
points.  A  very  attractive  feature  of  this  approach  is  that  it 
determines  discontinuities  and  tries  to  use  smoothness  only 
for  the  known  smooth  path  in  establishing  correspondence. 

The  determination  and  role  of  events  in  dynamic  scenes 
is  being  studied.  A  motion  event  may  occur  due  to  a  change 
in  the  motion  parameters  or  due  to  occlusion.  A  major 
thrust  of  other  research  is  to  develop  techniques  that  will 
recover  qualitative  information  about  depth  and  motion  of 
objects  using  constraint  propagation. 

3.6  Navigation 

Techniques  for  the  guidance  of  autonomous  mobile  robots 
or  vehicles  are  being  developed.  This  requires  techniques  to 
recover  information  about  the  environment  using  vision  and 
then  to  use  knowledge  based  techniques  to  control  the  navi¬ 
gation.  The  observation  that  the  location  of  road  vanishing 
points  does  not  change  significantly  from  frame  to  frame 
is  used  to  develop  an  algorithm  for  finding  road  boundaries 
[29],  The  algorithm  uses  the  hypothesise  and  test  paradigm. 
Results  of  this  approach  are  shown  in  Fig.  3. 


4  Range  Images 

The  long  term  goal  of  this  project  is  to  develop  techniques 
that  will  be  useful  in  object  recognition  and  navigation  using 
range  information. 

The  last  few  years  have  seen  increasing  attention  to  the 
analysis  of  range  images.  Range  images  may  be  obtained 
with  passive  methods,  such  as  binocular  stereo,  or  with 
range  sensors.  Range  images  contain  explicit  information 
about  surfaces.  This  explicit  information  facilitates  recog¬ 
nition  and  location  tasks  in  many  applications.  The  goal  in 
range  image  understanding  is  to  find  robust  symbolic  surface 
descriptions  that  are  independent  of  viewpoint.  Techniques 
to  characterize  surfaces  in  range  Images  are  being  developed 
5,4,241. 

Surfaces  are  segmented  using  local  features,  such  as  Gaus¬ 
sian  and  mean  curvatures  and  related  differential  geometric 
measures.  The  signs  of  the  curvatures  at  each  point  in  the 
image  are  used  to  assign  one  of  eight  basic  surface  types  to 
each  point  in  the  image.  The  next  step  is  to  develop  tech¬ 
niques  to  identify  surfaces  by  grouping  points  using  the  sur¬ 
face  type,  spatial  proximity,  and  other  criteria.  All  grouping 
processes  must  conform  to  the  sensed  information  since  the 
ultimate  truth  is  in  the  sensed  data.  This  stimulus  bound 
approach  uses  symbolic  surface  descriptions  in  the  '.^men¬ 
tation  of  images.  Figure  4  shows  a  result  of  this  segmenta¬ 
tion  approach.  The  symbolic  surface  descriptions  will  play 
important  role  in  many  applications. 

4.1  Recognition  Methods 

Object  recognition  is  a  major  motivation  in  most  image¬ 
understanding  systems.  Despite  strenuous  efforts,  only  lim¬ 
ited  success  has  been  achieved. 

The  object  recognition  task  can  be  classified  based  on 
difficulty  in  several  ways  One  way  is  based  on  the  degree  of 
uncertainty  allowed  in  the  object's  poeition  and  orientation. 
This  can  range  from  no  uncertainty,  to  uncertainty  only  in 
its  2-D  position,  through  uncertainty  in  both  its  2-D  posi¬ 
tion  and  rotation,  up  to  uncertainty  in  its  3-D  poeition  and 
orientation. 

The  task  can  be  further  classified  based  on  the  complex¬ 
ity  of  interactions  allowed  between  objects.  In  the  simplest 
case,  each  object  to  be  recognized  must  be  completely  visi¬ 
ble  and  surrounded  by  background.  Ln  a  more  complex  case, 
objects  are  allowed  to  touch  but  not  overlap.  In  tne  most 
general  case,  objects  are  allowed  to  partially  occlude  one 
another. 

Most  work  on  the  occluded-objects  problem  has  concen¬ 
trated  on  the  case  where  there  is  only  one  object  type  in  the 
scene.  Algorithms  developed  for  this  problem  can  be  ex¬ 
tended  to  cases  of  multiple  objects  types,  simply  by  running 
the  algorithm  multiple  times,  once  for  each  possible  object 
type.  This  is  not  ideal,  as  it  does  not  consider  or  take  ad¬ 
vantage  of  the  similarities  and  differences  among  possible 
objects.  Also  the  recognition  time  increases  linearly  with 
the  number  of  possible  objects,  after  a  fixed  time  for  pre¬ 
processing.  This  can  become  a  problem  as  the  number  of 
possible  object  types  increases. 


A  new  method  for  object  recognition  called  the  feature 
indexed  hypotheses  method  is  being  developed.  This  method 
breaks  the  recognition  process  into  two  phases:  hypothe¬ 
ses  generation  and  hypothesis  verification.  By  using  fea¬ 
tures  that  occur  multiple  times  in  the  possible  object  set, 
the  number  of  features  in  the  search  can  be  greatly  reduced 
generating  a  small  number  of  false  hypotheses  that  are  easily 
rejected.  By  carefully  selecting  the  features,  the  recognition 
time  growth  rate  can  be  reduced  to  the  square  root  of  the 
number  of  possible  objects.  This  method  also  has  the  advan¬ 
tage  that  unique  features  which  are  difficult  or  impossible  to 
find  if  the  possible  object  set  contains  many  similar  objects 
are  not  required.  The  method’s  feasibility  is  demonstrated 
by  the  results  of  a  prototype  two-dimensional  occluded- parts 
recognition  system. 

5  Knowledge-Based  Perception  Sys¬ 
tems 

Moat  complex  tasks  require  specialized  knowledge,  image 
understanding  is  no  exception.  The  last  few  years  have  seen 
an  increasing  application  of  knowledge  in  computer  vision 
systems.  The  application  of  knowledge  at  different  levels  in 
a  vision  system  is  being  studied. 

5.1  Knowledge-Based  Algorithms 

An  image  interpretation  system  is  a  program  that  derives  a 
scene  description  from  a  time-varying  image.  The  input  sig¬ 
nal  is  a  rectangular  grid  of  signal  samples  taken  at  regular 
time  intervals  f(x,y,t).  The  sample  can  be  taken  in  several 
frequency  ranges  (for  example,  red,  green,  and  blue)  and 
combined  into  a  time-varying  vector  field  C(x,  y,  t).  There 
is  wide  agreement  that  this  characterizes  the  environmental 
data  used  by  such  a  system  [9,17].  On  the  other  hand,  it  is 
much  more  difficult  to  formulate  a  description  of  the  output 
of  am  interpretation  system.  In  this  project,  the  output  of 
an  interpretation  system  is  a  network  representing  the  scene 
being  viewed,  an  approach  common  to  other  interpretation 
systems.  The  nodes  of  such  a  network  are  groups  of  objects 
within  the  scene:  individual  objects,  object  parts,  or  refer¬ 
ences  to  clusters  of  Image  events.  Arcs  in  the  network  are 
labeled  with  relations  between  the  objects.  Since  primitive 
object  parts  are  usually  depicted  as  geometric  solids  or  col¬ 
lections  of  joined  surface  patches  [10],  these  representations 
are  included  in  the  representation  used  in  this  work. 

However,  there  is  no  need  to  limit  ourselves  to  only  that 
type  of  representation.  In  fact,  the  study  of  which  types  of 
primitives  are  useful  and  how  they  relate  to  the  processing 
of  the  Image  is  part  of  our  current  research.  The  issues  of 
which  relations  to  use,  how  to  vary  those  relations  with  time, 
and  how  to  incorporate  the  variations  in  object  descriptions 
that  occur  over  time  are  also  being  studied. 

The  image  sequence  is  not  the  only  source  of  data  used 
by  an  interpretation  system.  A  large  database  of  relational 
and  descriptive  information,  including  information  about 
processes  and  procedures,  is  also  necessary  for  image  inter- 


pretation  [13,30],  These  data  are  a  symbolic  representation 
of  knowledge  about  the  world,  especially  as  that  knowledge 
pertains  to  the  interpretation  problem;  thus,  the  database  is 
usually  referred  to  as  a  knowledge  base.  For  example,  if  it  is 
known  that  the  camera  is  upright,  level,  and  outdoors,  then 
it  can  reasonably  be  expected  that  the  bright  area  at  the  top 
of  the  image  is  the  sky.  Such  simple  rules  and  other  more 
complex  inferences  form  the  greater  part  of  knowledge  for  in¬ 
terpretation.  The  problem  of  selecting  and  representing  the 
appropriate  knowledge  is  a  difficult  one.  Understanding  of 
image  interpretation  must  be  achieved  by  building  systems 
of  processes  that  work  in  restricted  domains.  In  building 
such  systems,  the  rules  and  processes  which  can  eventually 
be  incorporated  into  a  knowledge  base  are  learned.  The 
problem  then  is  to  organize  the  information  about  the  ob¬ 
ject  and  *he  "roeedur^T  fur  using  that,  informal ic a  is  that 
they  can  work  in  situations  where  the  answers  are  less  than 
certain. 

The  issues  are  basically  these: 

•  Organization  must  be  imposed  on  the  information  used 
for  image  interpretation,  because  that  information  is 
richly  detailed  and  complex,  and  because  large  amounts 
of  information  are  required. 

•  Varied  types  of  information  (for  example,  relations, 
procedures,  and  structural  descriptions)  need  to  be 
made  easily  accessible  at  many  levels  of  abstraction. 

•  There  must  be  a  way  to  control  the  selection  of  which 
information  is  considered,  which  programs  are  acti¬ 
vated,  and  what  information  is  passed  among  pro¬ 
grams. 

•  Interpretation  processes  need  to  deal  with  errors  from 
image  data  and  introduced  by  processes  producing 
partial  interpretation. 

•  All  components  of  the  interpretation  system  should  be 
able  to  deal  with  the  changing  nature  of  dynamic  data 
and  the  reorganization  of  interpretations  that  occur 
when  a  process  creates  new  hypotheses. 

6  Vision  Computer  Architecture 

The  real-time  application  of  image  understanding  algorithms 
requires  computer  hardware  that  allows  the  algorithms  to  be 
executed  at  high  speed.  The  NCUBE  machine  is  being  used 
to  study  hypercube  architectures  for  the  implementation  of 
vision  algorithms. 

This  research  is  concerned  with  developing  techniques 
to  perform  computer  vision  computations  on  loosely  and 
tightly  coupled  parallel  processors.  The  size  of  data  sets 
and  the  speed  at  which  they  can  be  created  in  current  prob¬ 
lems  :wamp  even  the  fastest  computers.  The  processing 
power  required  to  keep  up  with  the  input  is  greater  than 
100  MIPS.  This  level  of  processing  power  is  possible  only 
with  some  degree  of  parallelism.  However,  a  rule  of  thumb  is 
that  parallel  processors  can  rarely  sustain  an  efficiency  level 
of  greater  than  20 %  [38];  thus  the  machine  needa  to  have 
a  peak  power  of  about  500  MIPS.  Normally  the  number  of 


operations  far  exceeds  this  and  the  size  of  ihe  input  date 
set  is  often  much  greater  than  512  times  512  bytes,  either 
because  multiple  views  are  required,  or  because  the  sensor 
has  a  higher  resolution  (4096  times  4096,  for  example),  or 
more  than  one  modality  is  in  use. 

in  the  past,  vision  researchers  have  been  forced  to  use 
SIMD  meshes  (such  as  the  Goodyear  MPP  or  pyramids)  to 
achieve  this  rate  of  computation.  Hypercubes  offer  a  better 
alternative  than  SIMD  meshes  for  several  reasons. 

•  They  can  efficiently  simulate  these  SIMD  machines 

•  Their  interconnections  are  better  for  operations  such 
as  image  rotation,  that  are  inefficient  on  meshes  or 
pyramids 

•  The  hypercube  is  better  suited  to  higher  level  symbolic 
tasks  since  it  is  an  MIMD  machine,  the  nodes  have  far 
more  memory,  the  interconnections  are  much  better 
for  the  more  random  information  exchange,  and  the 
node  processors  have  a  better  instruction  set 

As  an  example  of  efficiency  in  vision  tasks,  a  hypercube 
with  p  processors  can  rotate  an  n  times  n  image  90°  in 
0  (nJ  log  p/p)  time,  versus  0  (»V y/p)  for  a  mesh  or  pyramid. 

Hypercube  architectures  for  MIMD  organizations  are 
better  than  architectures  with  SIMD  organizations  for  oper¬ 
ations  at  the  intermediate  levels  of  vision  processing  where 
the  sequence  of  operations  for  different  regions  in  an  im¬ 
age  are  usually  different  because  the  sequence  is  governed 
by  properties  of  the  region.  For  this  type  of  processing,  a 
loosely  coupled  multiprocessor  system  is  ideal.  If  dynamic 
scenes  are  involved,  then  massive  parallelism  is  essential  [lj. 

7  Sensor  Integration 

Representations  and  techniques  for  combining  vision,  range, 
tactile,  and  other  sensory  information  are  being  studied. 
Representations  that  will  allow  easy  inference  using  multi¬ 
sensors  are  not  known.  Representations  that  will  facilitate 
combining  multi-sensor  information  for  planning  and  recog¬ 
nition  tasks  using  partial  information  obtained  from  each 
source  are  key  to  the  success  of  sensor  fusion.  Techniques  to 
combine  uncertain  and  imprecise  information  are  being  de¬ 
veloped  using  nonmonotonic  and  probabilistic  approaches. 
Methods  that  try  to  combine  beliefs  and  disbeliefs  using  un¬ 
certainty  calculus  for  solving  problems  in  distributed  prob¬ 
lem  solving  systems  are  being  studied. 

Computer  vision  and  knowledge-based  systems  have  to 
deal  with  uncertain  and  imprecise  data  in  almost  all  phases 
of  reasoning.  Many  approaches,  such  as  Bayesian  methods, 
fuzzy  logic,  Dempster- Schaefer  theory,  and  qualitative  ap¬ 
proaches,  have  been  proposed  for  dealing  with  uncertainty. 
All  of  these  approaches  have  their  problems  and  advan¬ 
tages.  Unfortunately,  it  is  not  clear  which  method  is  best 
for  which  types  of  applications.  With  the  aim  to  under¬ 
stand  the  strenghts  and  weaknesses  of  each  approach,  a  lan¬ 
guage  called  ULOG  is  being  developed  that  will  implement 
uncertain  variables  in  a  PROLOG  type  environment.  The 
ULOG  language  allows  a  user  to  change  the  algorithm  used 


Figure  4:  A  range  image  and  its  segmentation  using  Besl’s 
and  Jain’s  algorithm  are  shown  in  the  top-left  and  bot¬ 
tom-right  corners  respectively.  Using  the  polynomial  de¬ 
scriptors  of  the  segmented  surfaces,  the  original  image  is 
reconstructed.  This  is  shown  at  top-right. 


Figure  5:  This  figure  shows  an  image  showing  the  classifi¬ 
cation  of  solder  joints  by  our  algorithm. 


Figure  6:  This  figure  shows  an  image  showing  "two  SEM 
images  of  a  section  of  a  wafer  and  the  3-D  surface  structure 
as  reconstructed  using  our  sem  stereo  algorithm. 


for  uncertainty  management  to  experiment  with  different 
approaches  in  order  to  find  the  best  approach  for  a  given 
problem. 

8  Applications 

Several  research  projects  are  closely  applied  with  specific  ap¬ 
plications.  Normally,  the  research  projects  are  focussed  on 
the  development  of  generic  algorithms  that  can  be  applied 
to  a  wide  variety  of  applications.  All  of  the  research  work 
on  generic  algorithm  can  lead  to  applications.  However, 
occasionally  some  application  is  investigated  in  the  Com¬ 
puter  Vision  Research  Laboratory  because  the  application 
provides  a  focus  for  learning  about  some  class  of  vision  al¬ 
gorithms.  Research  projects  that  are  closely  associated  with 
applications  are  discussed  in  the  following  sections. 

8.1  Solder  Joint  Inspection 

This  project  is  for  improving  the  reliability  of  solder  joints 
and  to  reduce  the  cost  of  manufacturing  by  reducing  the 
fault  rate.  Techniques  are  being  developed  for  inspection  of 
solder  joints  to  identify  not  only  good/bad  joints  but  also  to 
identify  the  nature  of  defects.  By  identifying  the  nature  of 
defects  and  using  trend  analysis,  the  system  may  be  able  to 
indentify  the  process  parameters  that  may  result  in  defective 
joints  (7).  An  algorithm  based  on  surface  characteristics 
classifies  the  joints.  The  performance  of  the  current  version 
of  the  algorithm  is  97-98%  correct  results.  A  knowledge- 
based  approach  to  the  task  is  being  implemented  to  improve 
the  performance  of  the  classifier  still  further  and  to  identify 
process  parameters  [3).  Some  results  from  this  project  are 
shown  in  Figure  5.  Recent  extensions  to  this  research  project 
include  inspection  algorithms  for  circuit  boards  with  surface 
mounted  components. 

8.2  Semiconductor  Wafer  Inspection 

This  project  is  a  part  of  the  center  of  excellence  in  semi¬ 
conductor  manufacturing.  The  project  is  concerned  with 
the  in-process  inspection  of  semiconductor  wafers  to  iden¬ 
tify  defects.  The  objective  is  to  identify  defects  to  control 
subsequent  processes;  if  possible,  the  system  will  take  cor¬ 
rective  action  and  adjust  the  defective  process  parameters 
at  the  stage  that  is  be  responsible  for  the  defect.  The  cor¬ 
rective  actions  are  automatically  performed  by  an  expert 
system  that  controls  the  processes.  The  inspection  is  done 
with  scanning  electron  microscopy  (SEM)  and  optical  mi¬ 
croscopy.  Current  activities  include  the  study  of  SEM  im¬ 
ages  to  implement  stereo  approaches  for  those  images  to  per¬ 
form  analysis  of  those  images  to  inspect  the  nature  of  edges 
and  other  features  on  a  wafer.  A  model-based  stereo  algo¬ 
rithm  has  been  implemented  to  reconstruct  the  3-D  struc¬ 
ture  of  a  wafer  section.  Figure  6  shows  3-d  surface  structure 
recovered  by  our  algorithm.  Knowledge-based  controls  are 
being  developed  for  the  interface  between  the  vision  sys¬ 
tem  and  the  expert  system  in  the  automated  semiconductor 
manufacturing  project. 
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Abstract 

Congress  has  displayed  substantial  interest 
in  accelerating  the  dissemination  of  advanced 
automation  technology  throughout  United  States 
industries.  The  Space  Station  was  selected  as  the 
high-technology  program  to  serve  as  a  hLghly  visi¬ 
ble  demonstration  of  .advanced  automation,  and  spur 
dissemination  of  the  technology  to  the  private 
sector.  NASA  has  recently  initiated  an  Automation 
and  Robotics  Program  to  serve  as  the  principal 
Research  and  Technology  program  contributing  to 
Space  Station  automation. 

The  potential  benefits  of  autonomy  in  terms 
of  increased  capability,  safety,  reliability, 
efficiency,  and  cost  savings  for  operational  sys¬ 
tems  on  the  Space  -tation  has  given  strong  impetus 
to  the  development  of  a  Project  for  feasibility 
demonstrations  of  automation  technology.  Addi¬ 
tional  strong  motivation  was  given  by  the  report 
and  recommendations  to  Congress  by  the  Advanced 
Technology  Advisory  Committee  (ATAC):  "The  devel¬ 
opment  of  the  Space  Station  offers  a  ’hance,  both 
to  advance  the  technology  of  automat,  and  robot¬ 
ics  as  proposed  by  Congress  and  to  put  the  tech¬ 
nology  to  use.  The  use  of  advanced  automation  and 
robotics  technology  in  the  Space  Station  would 
thereby  provide  a  logical  driving  force  for  a  new 
generation  of  machine  intelligence,  robotics, 
computer  science,  and  microelectronics." 

The  NASA  Systems  Autonomy  Demonstration  Proj¬ 
ect  has  been  initiated  in  response  to  this  Con¬ 
gressional  interest  and  ATAC  recommendations  for 
Space  Station  automation  technology  demonstra¬ 
tions.  The  demonstrations  will  begin  with  the 
testbed  for  the  Space  Station  Then-ial  Control 
System  (TCS)  in  1988.  Additional  demonstrations 
are  scheduled  for  1990,  1993,  and  1996.  The  1990 
demonstration  will  involve  coordinated  control  of 
two  Space  Station  subsystems  through  cooperating 
expert  systems,  the  1993  demonstration  will 
involve  hierarchical  automation  of  multiple  sub¬ 
systems,  and  the  1996  demonstration  will  involve 
distributed  automation  of  multiple  subsystems. 

The  1988  TCS  Demonstration  will  be  a  joint 
cooperative  effort  between  NASA  Ames  Research 
Center  (ARC)  and  NASA  Johnson  Space  Center 
(JSC).  Knowledge-engineering  and  operator- 
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interface  technologies  for  Systems  Automation  will 
be  developed  by  knowledge  engineers,  AI  research¬ 
ers,  and  human  factors  researchers  at  ARC  by  rely¬ 
ing  on  a  close  working  relationship  with  the 
domain  experts,  knowledge  and  integration  engi¬ 
neers,  and  mission  operations  personnel  at  JSC. 

This  paper  provides  an  overview  of  the  NASA 
Systems  Autonomy  Demonstration  Project,  and 
describes  general  details  of  the  1988  TCS  Demon¬ 
stration  including  expert  system  functional  objec¬ 
tives,  system  concept,  and  development  plans. 


Introduction 

In  keeping  with  the  mandates  of  the  Nationa. 
Aeronautics  and  Space  Act  of  1958  and  the  National 
Space  Strategy  approved  by  the  President  and  Con¬ 
gress  in  1984,  NASA  has  set  for  itself  a  major 
goal  of  "conducting  effective  and  productive  space 
applications  and  technology  programs  which  con¬ 
tribute  materially  toward  U.S.  leadership  and 
security." 

The  Report  of  the  National  Commission  on 
Space  (May  1986)  in  its  vision  of  the  next 
50  years  on  space  strongly  recommends  an  integra¬ 
tion  of  humans  and  machines  through  automation  and 
robotics.  Specifically  it  is  recommended  that 
"NASA  explore  the  limits  of  expert  systems,  and 
tele-presence  or  tele-science  for  remote  opera¬ 
tions,  including  ties  to  spacecraft  and  ground 
laborator  ies . " 

Congress  has  displayed  substantial  interest 
in  accelerating  the  dissemination  of  advanced 
automation  technology  within  United  States  indus¬ 
tries.  Space  Station  was  selected  as  the  high- 
technology  program  to  serve  as  a  highly  visible 
demonstration  of  advanced  automation,  and  spur 
dissemination  of  the  technology  to  the  private 
sector.  NASA  has  recently  initiated  an  Automation 
and  Robotics  Program  to  serve  as  the  principal 
Research  and  Technology  program  contributing  to 
Space  Station  automation. 

The  potential  benefits  of  autonomy  in  terms 
of  increased  capability,  safety,  reliability, 
efficiency,  and  cost  savings  for  operational  sys¬ 
tems  on  the  Space  Station  has  given  strong  impetus 
to  the  development  of  a  Project  for  feasibility 
demonstrations  of  automation  technology.  Addi¬ 
tional  strong  motivation  was  given  by  the  report 
and  recommendations  to  Congress  by  the  Advanced 
Technology  Advisory  Committee  (ATAC):  "The  devel¬ 
opment  of  the  Space  Station  offers  a  chance,  both 
to  advance  the  technology  of  automation  and  robot¬ 
ics  as  proposed  by  Congress  and  to  put  the 


1 


technology  to  The  iso  o:'  tz.  meed  lutonat -or. 

»r.d  robot  ics  technology  in,  tr.>»  It  o-'  It. jt.cn  wou.J 
thereby  provide  a  logica-  dri^;”g  force  for  i  new 
generation  3:'  machine  intelligence,  ricotics,  com¬ 
puter  science,  and  microelectronics."' 

The  NASA  Systems  Autonomy  Temonst r  it  ion  Pro;- 
eot  nas  been  initiated  in  response  to  tn.is  Con¬ 
gressional  interest  and  AT A>1  reeommer  :at  10ns  :'or 
Space  Station  automation  technology  lemcnstra- 
tiors.  rr.e  ierorstrat  ions  w  1  _  1  begin  with  the 
testned  for  tr.e  Space  Station  thermal  control 
system  ! TCS )  in  '988.  Additional  demonstrations 
ire  scheduled  for  '990,  '993.  i"d  ’99o.  The  '990 
demonstration,  will  involve  coordinated  control  of 
two  Space  Station  subsystems  through  cooperating 
expert  systems,  the  '993  demonstration  will 
involve  hierarchical  automation  of  multiple  sub¬ 
systems,  and  the  '996  demonstration  will  involve 
distributed  automation  of  multiple  subsystems. 

The  1 2 3 4 938  TCS  Demonstration  will  be  a  joint 
cooperative  effort  between  NASA  Ames  Research 
Center  (ARC)  and  NASA  Johnson  Space  Center 
l JSC).  Knowledge-engineering  and  operator- 
interface  technologies  for  Systems  Automation  will 
be  developed  by  knowledge  engineers,  A!  research¬ 
ers,  and  human  factors  researchers  at  ARC  by  rely¬ 
ing  on  a  close  working  relationship  with  the 
domain  experts,  knowledge  and  integration  engi¬ 
neers,  and  mission  operations  personnel  at  JSC. 

The  TCS  Automation  involves  the  implementa¬ 
tion  of  current  AI  technology  Into  the  real-time 
dynamic  environment  of  a  complex  electrical- 
mechanical  Space  Station  system.  The  TCS  includes 
real-time  nominal  control,  fault  diagnosis  and 
correction  of  real-time  problems,  design  and 
reconfiguration  advice  on  the  thermal  testbed 
(TTB),  and  an  intelligent  interface  to  both  novice 
and  expert  users.  The  TCS  Demonstration  will 
accelerate  the  transfer  of  Systems  Autonomy 
research  technologies  to  user  applications  in  a 
real-time  operational  environment,  and  increase 
user  confidence  in  the  new  technologies. 

Systems  Autonomy  Demonstration  (SAPP) 

Pro  ject 

Ob  ject  ives 

The  objectives  of  the  Systems  Autonomy  Demon¬ 
stration  Project  are  to  provide: 

1.  Technical  base  of  in-house  personnel  and 
development  tools  to  facilitate  AI  tech¬ 
nology  transfer 

2.  Technology  focus  for  Automation  Research 
and  Development  in  support  of  NASA's 
Space  Programs 

3.  Means  for  validation  and  demonstration  of 
Automation  Technology  prior  to  transfer 
to  Agency  programs 

4.  Credibility  of  Automation  Technology 
within  NASA 


0.  Credibility  of  NASA  A I  expert,  s*.  :  ;  t.-.e 
outside  A I  community 

Broad  Approach 

The  Systems  Autonomy  Demons tr  it  i or;  <  ,;a.'  f- . 
will  be  a  joint,  cooperative  effort  between 
research  and  operational  NASA  Centers.  The 
required  AI  technologies  w: i i  be  developed  and 
implemented  by  knowledge  engineers,  3rd  A!  and 
human  factors  researchers  at  ARC;  while  relying 
upon  the  domain  experts,  know,  edge  and  Integra'- .on 
engineers,  and  mission  operations  personnel  res. J- 
mg  at  operational  NASA  Centers. 

The  SADP  project  approach  will  involve  a 
mul t idiscipl inary  integration  of  knowledge  engi¬ 
neering,  man/machine  interfaces,  and  systems 
architecture  to  enhance  automation  of  the  Space 
Stat  ion . 

each  project  demonstration  will  proceed 
through  a  phased  knowledge  engineering  methodology 
consisting  of:  prototype  knowledge  base  develop¬ 
ment,  incremental  knowledge  base  expansion,  paral¬ 
lel  operator  interface  development,  and  implemen¬ 
tation  in  a  realistic  environment.  Initial  AI 
system  development  will  be  carried  out  m-house  so 
as  to  develop  a  strong  tecnnical  base  within  NASA. 

Demonstrations  will  involve  interaction  with 
both  experts  and  novice  personnel  representing 
mission  operations,  automated  flight  subsystems, 
and  automated  sciences.  This  interaction  with 
astronauts  and  ground  crew  is  a  critical  component 
of  the  project  to  ensure  that  the  human  interface 
(man-machine)  issues  are  properly  addressed. 

SADP  Demonstration  Selection  Criteria 

The  following  criteria  will  be  used  in 
selecting  demonstrations: 

1.  Provide  maximum  use  of  existing  AI 
technologies 

2.  Illustrate  gains  in  human  productivity 
and  reductions  in  manpower  requirements 
resulting  from  automation 

3.  Have  access  to  domain  experts 

4.  Requires  no  unattainable  personnel  and 
equipment  resources 

5.  Leave  a  framework  of  people  and  tools 
which  will  facilitate  future  technology 
transfer 

6.  Transfer  developed  technology  to  the 
Space  Station 

SADP  Management 

The  information  Sciences  and  Human  Factors 
Division  of  NASA's  Office  of  Aeronautics  and  Space 
Technology  provides  overaLl  direction,  funding, 
and  evaluation  of  the  Systems  Autonomy  Demonstra¬ 
tion  Project  being  managed  by  the  Systems  Autonomy 
Demonstration  Project  Office  in  the  Information 
Sciences  Office  at  ARC. 
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' -S  Selection  Ratior.a.e 


.ovi.oprr:i?:;t  arid  S.muiatio1 
the  Mias  ion  Operations  Or 


The  rationale  for  select  i-  g  '  O  is  :>-e  r~o 
str  it  lor.  Project  :.»  th  it  it  meets  the  move  iACP 
criteria  arc  orovides: 


' .  Guarantees  access  i;  available  to  Domain 

wAp^r'ti 

2.  TCS  slow  dynamics  reduce  technca.  risks 

3.  Adequate  personnel  and  resources  are 
avai labie 

4.  Dencnstrat  ior.  jcnedtle  matches  we.l  -.itr. 
TTB 

5.  Environment  for  interface  with  Space 
Station  Data  Management  Testbed 

TCS  Demonstration  Technical  Objectives 

Technical  objectives  of  TCS  Autonation  are  to 
demonstrate : 

Real-time  nominal  cents oi  ind  reconfigu¬ 
ration  for  four  to  five  failure  nodes 
.  Fault  diagnosis  of  25  to  30  major  fail.  . re 
modes 

3.  Trend  analysis  incipient  fa: .are 
prevent  ion 

4.  Intelligent  interface  to  both  novice  and 
expert  users 

5.  Design  advice  on  TTB 

6.  Training  assistance 


Technoloc 


The  ma^or  teennoiogy  thrusts  of  the  TCS 
Demonstration  are: 


TCS  1-Ti‘T  i ,  Appro  acn 

The  Space  St  it  ion  TTB  is  ce.’V  :  ■■  : 

the  Crew  and  Thermal  Systems  11 .  v  i  i.on  at  1 
-.rich  the  domain  expertise  is  cei-g  provided, 
knowledge  engineering  and  demons  tr.a  t  .on  prototype 
development  are  being  done  oy  tne  SwD?  Office  in: 
the  Artificial  Intelligence  fesearen  Hr  cc-  „ . ;  - 
support  from  the  Aerospace  Hunan  Factors  - 

at  ARC.  The  Systems  Development  ;rd  S.mu.it;:n 
Division  at  JSC  provides  support  arc  participates 
with  the  SADP  Office  at  ARC  in  the  know. edge  eng: 
r.eering  and  expert  system  development  aspects  ;f 
tne  TCS  project  and  provides  testbed  .rtegri- 
tion.  The  Mission  Operations  Directorate  ;f  JSC 
provides  consultation  ana  advice  or  recent  trend.; 
and  technology  advancements  in  operations'  a_t  :.-s 
t  ion  and  the  application  of  these  tecn.no.  ig.es  a: 
current  mission  operations'  philosophy  to  tne 
TCS.  The  general  TCS  approach  is  sr.own  . n  rig.  " 

Fac i i i t les 

The  major  facility  required  for  the  TCS 
Demonstration  is  the  TTB  being  constructed  at 
JSC.  The  TTB  includes  the  following  subsystems: 

1)  Thermal  System  Test  Articles  (pumps,  radiators 
evaporators,  condensers,  busses),  and  2)  a  Tata 
Acquisition  and  Control  Ssytem. 

A  facility  is  located  at  ARC  for  development 
of  the  Thermal  Expert  System  (TEX3YS).  This 
facility  will  consist  of  1)  AT  HW/SW  development 
tools  and  2)  a  simulation  HW.  model  of  the  TTB 
for  TEXSYS  development  and  validation. 


Integration  of  knowledge  base  systems 
into  a  real-time  environment 

2.  Causal  modeling  of  complex  components  and 
elements 

3.  Combining  model-based  and  experiential 
knowledge  for  diagnosis 

4.  Trend  analysis  heuristic  ruies 

5.  A I  validation  methodologies 

TCS  Automation  Benefits 

The  major  benefits  of  TCS  Automation  are: 

1.  Reduces  need  for  crew  and  ground  monitor¬ 
ing  of  TCS 

2.  Increases  crew  safety  through  improved 
systems  monitoring 

3.  Provides  TCS  design  assistance 

4.  Simplifies  novice-  and  expert-user 
training 

TCS  Management  Organization 


TCS  Schedule 

The  broad  overall  TCS  Demonstration  Project 
Schedule  is  shown  in  Fig.  2 

Success  Criteria 

Programmatic  success  criteria,  althougn  more 
difficult  to  define  than  technical  success  cri¬ 
teria  (meeting  system  specifications),  is  of  equa 
importance.  These  criteria  are  the  incorporation 
of  systems  autonomy  technology  (developed  as  a 
result  of  and  demonstrated  during  the  TCS  Demon¬ 
stration)  in  various  Space  Station  subsystems  and 
systems.  This  criteria  does  not  imply  direct 
mcorporat ion  of  TEXSYS,  or  any  part  thereof. 
Rather,  it  implies  an  influence  on  Space  Station 
Project  Offices,  measured  by  the  incorporation  of 
autonomy  requirements  in  subsystem  requirements 
documents  and  the  inclusion  of  automation  in  the 
design  and  development  of  those  subsystems. 


The  Thermal  Control  System  (TCS)  Demonstra-  Discussion 

tion  Project  is  managed  by  the  SADP  Office  at  ARC 

m  a  close  working  relationship  with  Aerospace  This  section  discusses  the  TCS  System  Con- 

Human  Factors  Division  at  ARC,  the  Crew  and  Ther-  cept.  Specific  Technical  Objectives,  Development 

ma i  Systems  Division  at  JSC,  the  Systems  Plan,  and  Status. 
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-L.ec.:'ic  Thermal  Control  Expert  System  Objectives 

The  Thermal  Control  Expert  System  1 TEX3VS ) 
xr.cw  .edge-based  functions  within  the  ECS  ire: 

'.  Diagnosis  and  Paul  -.-Correct  ion  Advice 

T .  Incipient  Failure  Prevention  Through 
Trend  Analysis 

j.  Nominal  Real-Time  Control  and  F.au.t 
Correct  ion 

Intelligent  Interface  to  Novice  and 
Expert  users 
Training  Assistance 

6 .  Des ign  Assistance 

Diagnosis  and  Fault  Correction  Advice .  The 
DCS  tin  have  faults  in  three  major  categories: 
components,  control  malfunctions,  and  sensor  mal¬ 
functions.  The  TCS  demonstration  will  show 
expert-level  anility  to  diagnose  and  suggest  cor¬ 
rective  actions  cn  approximately  2 5  to  30  common 
TCS  faults,  representing  essentially  ail  major 
~odes  of  TCS  failures. 

Incipient  Failure  Prevention.  Human  beings 
me  -otcr ; 0:.s  1  y  poor  »  the  slow  and  careful  anal- 
y  s  i mat  is  needed  to  prevent  low  frequency 
dynamic  anomal.es  from  escalating  into  problems. 

A  potential  strength  of  a  knowledge-based  systems 
iSprouch  to  thermal  management  is  the  use  of  trend 
analysis  to  detect  long-term  degradation  and 
-ecorf igurat  ion  as  required  to  prevent  system 
parimeters  from  exceeding  operational  limits.  The 
demonstration  will  exhibit  "offline"  (i.e.,  during 
rorcrisis  times)  analysis  of  trends  to  detect 
me—  alous  values  to  make  corrections  to  the  ther- 
mal  system  before  serious  problems  result. 

sea_.-t  :me  looted  and  Fault  Correction.  The 
-stration  will  exhibit  real-time  nominal  con¬ 
'd  ..  is  w»il  as  real-time  correction  of  at  least 
fs ir  or  five  major  failure  classes  of  the  thermal 
ivitem.  In  t.ne  context  of  the  thermal  system, 
'“al-t.mp  u  a  matter  of  seconds.  The  TCS  expert 
■'/item  w:  .  i  analyze  actual  sensor  data,  notice  and 
!•  ignose  problems,  and  correct  (or  bypass)  prob- 

ny  sending  control  signals  to  the  thermal 
; y  s  o  em 


Intelligent  Interface.  Tr.e  Jemorutr  it  .or. 

«...  snow  tne  ioil:ty  of  trie  Crow  I  w  ige-i.  au-sd  T 
expert  system  to  explain  its  reasoning  to  .sirs. 
The  operator  interface  wii.  ai.ow  is-ms  icons.;  t  a 
information  on  testbed  schematics,  ill  stages  of 
f.au.t  reasoning,  basic  physical  principles  under¬ 
lying  component  and  TCS  system  behavior,  and  pro¬ 
vide  guidance  m  making  decisions  involving  ther¬ 
mal  management.  The  interface  will  oe  a  "direct 
manipulation”  style  interface,  combining  mouse- 
based  pointing  and  menu  selection  is  user  input; 
and  the  interface  will  show  some  degree  of  under¬ 
standing  of  the  skill  level  of  its  user. 

Training  Assistance.  A  beneficial  side 
effect  of  knowledge-based  systems  is  that  the 
knowledge  bases  have  substantial  utility  for 
future  training  purposes  with  the  system.  The 
information  display  capabilities  will  demonstrate 
now  the  knowledge-based  TCS  expert  system  car,  be 
used  for  purposes  of  crew  and  ground  operator 
training  in  the  context  of  Space  Station. 

Tra.nees  will  be  able  to  examine  data  ar.d  simulate 
the  effects  of  all  known  faults. 

Design  Vsslstar.ee.  The  expert  ..vs ton 
modeling  and  simulation  knowledge  base  provides  ■ 
substantial  capacity  for  intelligent  assistance  ti¬ 
the  design  engineer  using  the  TTB.  The  informa¬ 
tion  and  display  capabilities  will  demonstrate  the’ 
ability  to  automatically  reflect  new  pnysical 
realities  resulting  from  design  changes  during 
system  configuration  change  investigations. 

TEXSYS  Conceptual  Configuration  with  the  TTB 

The  conceptual  configuration  of  TEXSYS  within 
the  TTB  is  shown  in  Fig.  4. 

Individual  test  articles  are  directly  con¬ 
nected  to  control  computers  -ImicroVAXes )  which 
then  connect  to  an  Ethernet.  A  DACS  computer  (a 
larger  microVAX  II)  acts  as  a  system  controller, 
central  data  router,  and  command  queuer  for  the 
TTB.  The  TEXSYS  initially  running  on  a  special¬ 
ized  LISP  machine  will  be  connected  via  standard 
DECnet  protocols  to  the  Ethernet.  The  machine  may 
receive  data  from  the  DACS  system  and  pass  com¬ 
mands  to  the  DACS  system.  If  this  routing  strat¬ 
egy  is  not  sufficiently  fast  or  powerful,  data 
will  be  received  from  and  commands  passed  directly 
to  the  test  article  controllers. 

Another  possibility  for  increasing  speed 
would  be  the  use  of  a  conventional  computer  as  a 
front-end  processor  (FEP)  along  with  a  LISP 
machine.  Such  an  arrangement  could  be  useful  if 
the  major  speed  bottleneck  turns  out  to  be  in 
handling  the  raw  data  from  DACS.  The  FEP  could 
handle  preprocessing  tasks  to  reduce  the  raw  data 
to  a  compressed  amount  of  information  that  can  be 
handled  by  the  expert  system.  It  might  be  possi¬ 
ble  that  the  DACS  itself  could  provide  this  data 
reduction.  Further  investigation  of  this  area 
will  be  necessary . 
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tion.  SyTe  .  .  c.;  il.  isu  env 1 ronme"t  provides 

j.mOiO  mechanisms  to  l’.reo'..y  oil!  the  FORTRAN 
suDrout ;  nes  that  ore  used  to  send  the  proper 
information  requests  and  r  iconfig  int  ion  commands 
out  to  subsystem  controllers  via  the  Ethernet. 

TEXS 1  5  <nowLe:i£e  Base 

the  TEXSYS  Knowledge  oase  will  rely  to  a 
.urge  decree  on  noth  experiential  heuristic  rules 
»nel  causal  or  model-based  reasoning.  The  initial 
TEXSYS  concept  w: • 1  use  a  Y- one -rased ,  hierarchi- 
■i..  ot>  vet -or  :->r.ted  representation  of  knowl¬ 
edge  .  -rime-oasej  -ears  that  oacn  structural 
comoorent  of  a  tr.erm.U  ivsT^m,  or  each  class  of 
components,  .s  represented  oy  a  collection  of 
quantitative  ir.d  qualitative  facts  3 prut  the  com¬ 
ponent.  Hierarchical  means  that  each  entity  is 
not  represented  is  a  separite  item,  but  as  a  tree 
of  struct  jres.  Object -or i ented  means  tnat  both 
factual  and  procedural  krowledje  ire  accessed 
through  -re  same  mechanisms.  Figure  r  shows  exam¬ 
ples  or"  now  the  knowlec0e  base  is  suodivded  into 
thermal  rules,  systems,  and  component,  models. 

TEXSYS  Model: ng  and  Simulation 

The  basis  of  ail  modeling  and  simulation  in 
TEXSYS  is  tne  structural  and  functional  knowledge 
b ise .  Part  of  the  knowiedg^-based  construct' on 


tasK  is  to  mclud®  pointers  to  relevant,  existing 
mathematical  models,  to  first  principles  of  ther¬ 
mal  engineeri-g,  and  to  heuristics  for  parameter 
propagation,  for  ail  components  and  subsystems 


involved  in  thermal  management.  Simulation  m  any 
of  those  cases  proceeds  in  a  straightforward, 
object-oriented  manner.  This  means  that  a  "simu- 
1 ite  yourself"  message  gets  passed  to  the  re’evant 
atructure  that  is  tu  00  modeled  and  procedui a  1 
know* edge  of  the  appropriate  form  is  activated. 

For  qualitative  mode, 3,  this  process  is  normally  a 
.■a.,  to  1  FORTRAN  subroutine;  however,  for  quali- 
<  itive  causal  models,  software  tool0  convert  laws 
nf  thermal  sciences  into  act  10  is;  and  for  heuris¬ 
tic  propagation  of  parameters,  forward  chaining  is 
•  orna  1  1  y  ;a  t ;  ;  r  10 1  1 1  y  . 

The  me  ;t  difficult  technical  task  will  be  the 
third  step  <  the  first  two  oeing  description  of  the 
..it  ructura :  and  functional  Knowledge,  and  develop- 
,rig  .  ;fer<'nce  me-rods  t  *  *  use  that  knowledge). 

Thi.;  third  ti;/  is  selection  of  whicn  of  the  three 
types  of  mod'1.  1  'heuristic,  qualitative,  or  quan- 
t  :  t  ativej  to  use  lor  any  simulation,  ind  to  select 
the  ippronr  ;  it.'  combination  01  information  from 
different  models.  The  selection  itself  will 


ilmost  jjrelv  oe  neuris’ic,  c .  ,,'d  .  i:  ■'  -./pert 
knowledge  of  the  relevance  .ml  trustwor  rr. .  ; 

the  var.ous  types  of  models  in  different  situa¬ 
tions.  e.-mio  mat  ions  of  models,  especially  wrier  w-- 
are  lttemptirsg  to  combine  quantitative  ,nd  quali¬ 
tative  Knowledge,  will  to  1  sign  1  !':■-■  ini  rose  iron 
task.  Most  of  the  work  will  re  experimental, 
testing  various  models  in  many  d  1  Yf.'i  t  3  tua- 
tions  and  determining  the  relevant  steed  iccu:  icy' 
cost  tradeoffs  that  apply. 

General  Development  Plan 

!n  any  knowledge  engineering  project  the  work 
proceeds  by  incremental  refinement  of  a  relatively 
simple  system,  adding  knowledge  and  consequently 
ability  for  increased  performance;  and  oy  carrying 
out  research  in  how  to  better  combine  types  of 
knowledge  and  reasoning  methodologies. 

As  one  of  the  largest  knowledge  engineering 
projects  yet  attempted,  this  demonstration  * : . 1 
use  the  previously  described  approacn,  and  wi ; . 
also  proceed  along  traditional  project  development 
methods:  definition  of  tne  problem,  spec  1  f: eat . an 
of  system  requirements,  definition  of  system  spe¬ 
cifications,  development,  validation,  integration, 
checkout,  and  demonstration. 

Specific  Development  1'ian 

The  development  and  demonstration  of  the 
TEXSYS  system  will  be  accomplished  through  six 
major  stages,  most  of  which  are  separated  by  ma  'or 
project  reviews.  These  stages  include: 

1.  Prototype  Development  Stage 

2.  Requirements  Definition  Stage 

3.  System  Specification  Stage 

4.  Initial-System  Development  Stage 

5.  Final -System  Development  Stage 

6.  Demonstration  Stage 

Prototype  Development  Stage.  As  a  first  step 
in  thp  problem  definition  and  incremental  engi¬ 
neering  process,  a  small  but  significant  prototype 
was  constructed  in  June  and  July  of  1986.  The 
objectives  of  the  prototype  development  were: 

1.  To  learn  significantly  more,  directly 
from  an  expert,  about  the  TTB  environment 
and  about  thermal  engineering,  especially 
as  related  to  two-phase  thermal  systems 
on  Space  Station 

2.  To  provide  knowledge  engineering  training 
for  ARC  RI  SADP  personnel  in  a  practical, 
problem-oriented  environment 

3.  To  build  a  working  prototype  system  that 
would  serve  as  a  starting  point  for 
future  work 

An  analysis  was  made  of  probable  TEXSYS  func¬ 
tional  and  performance  requirements,  -vailable 
hardware,  software,  expert-system  building  tools, 
and  training  and  engineering  support.  Based  on 
this  analysis,  a  selection  was  mad..  as  to  the 
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ion  Stage.  The 

next  step  in  the  development  and  demonstration  : : 
tr.e  TEXSYS  system  is  the  formal  and  specific  def;  - 
ration  of  requirements.  Particular  care  will  be 
paid  to  interfaces  to  the  operator,  to  the  TCS 
Testbed  computers,  to  real-time  data  collection 
and  TEXSYS  oerformance  requirements,  and  to  the 
arch i tectur a .  structure  of  the  TEXSYS  knowledge 
case.  These  features  will  be  documented  in  tne 
TEXSYS  System  requirements  Definition  and  w:.,  -•» 
reviewed  it  tre  TEXSYS  System  Requirement.;  Rev ; .••„ . 

System  Specification  Stage.  Following  “he 
successful  jcccmp. isnment  of  the  SRR,  work  w: . . 
shift  to  the  generation  of  a  system  design, 
including  design  of  the  knowledge  base  architec¬ 
ture.  specific  interfaces  with  necessary  utili¬ 
ties,  other  systems,  and  the  human  operator  or 
user  of  the  TEXSYS,  and  the  structure  and  format 
of  data  to  oe  used  as  real-time  input  and  out¬ 
put.  This  stage  will  be  documented  in  the  TEXSYS 
System  Design  Specification  which  will  aiso  spec¬ 
ify  the  delivery  hardware  and  software.  This 
stage  is  completed  at  PDR. 

Initial  System  Development  Stage .  The  ini¬ 
tial  TEXSYS  development  activities  will  consist  of 
procurement  of  test  and  demonstration  hardware  ir.d 
software,  and  include  two  major  phases  of  knowl¬ 
edge  base  development.  The  development  activities 
it  ARC  will  concentrate  on  development  of  TEXSYS 
knowledge  bases  and  the  human  interface  to  TEXSYS, 
while  JSC  will  take  the  lead  in  developing  the 
software  needed  to  integrate  and  interact  with  the 
reai-time  systems  with  which  the  TEXSYS  will 
interface . 

Phase  one  of  the  knowledge  base  development 
will  consist  of  the  acquisition  and  organization 
of  knowledge  about  the  TCS  Testbed  components  and 
topology,  and  the  development  of  rules  for  detect¬ 
ing  and  diagnosing  problems.  During  this  phase  a 
static  knowledge  base  will  be  used  for  testing 
purposes . 

Phase  two  will  include  the  modification  of 
the  system  to  successfully  accommodate  real-time 
operation  and  the  provision  of  simulated  dynamic 
data  to  test  this  major  enhancement.  This  stage 
13  complete  at  CDR. 

Final  System  Development.  After  completion 
of  the  CDR  and  delivery,  installation,  and  check¬ 
out  of  the  TCS  Demonstration  software  at  JSC,  the 
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Fm.aiiy,  the  verification  and  validation 
tests  sea.  I  be  performed  on  the  final  demonstra¬ 
tion  TTB  configuration  jointly  by  ARC  and  JSC  to 
ensure  tn-t  the  expert-system  knowledge  base  .s 
comp.rte  inc  correct.  After  passing  these  tests, 
TEXSYS  shall  be  considered  ready  for  the  SADR  '  i?,o 
demonstration  phase. 

This  phase  of  the  TEXSYS  development  will 
conclude  with  the  TCS  Operational  Readiness  Review 
(ORR).  The  ORR  will  examine  all  TMS  Demonstration 
activities  to  determine  the  readiness  of  the  sys¬ 
tem.  the  procedures  and  documentation,  and  the 
personnel  for  the  conduct  of  the  operational  phase 
of  the  TCS  Demonstration. 

TCS  Demonstration  Stage.  After  successful 
completion  of  the  ORR,  the  final  phase,  the  Demon¬ 
stration  Phase,  will  begin.  This  stage,  conducted 
jointly  by  ARC  and  JSC,  will  include  the  demon¬ 
stration  operations,  and  the  post  demonstration 
analysis  and  review. 

The  operations  stage  will  involve  the  actual 
conduct  and  documentation  of  the  TEXSYS  in  manage¬ 
ment  and  control  of  the  TCS  Testbed.  The  analysis 
and  review  phase  will  provide  an  integrated  retro¬ 
spective  inalysis  of  the  system  capabilities,  and 
the  development  process,  to  provide  insight  into 
the  effectiveness  of  the  TEXSYS  in  management  and 
control  of  the  TCS  Testbed  anc  to  identify 
improvements  that  can  be  made  in  later  phases  of 
the  SADP  project  activities. 
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The  : r: :  p  :  .1  i  demonstration  in  ' -)88 
iutoir.it  ion  of  the  Space  Station  Thera .  T  ;r 
System.  An  .niti.il  expert  system  prototype 
seen  dive. oped  it  ARC  and  the  development  : 
.pressing  on  schedule  for  a  demonstration  or. 
Thermal  Testbed  at  JSC  in  August  '983. 
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Opera: : mil  Readiness  Review  (ORR-7  33) 
TCS  demonstration  Review  (TDR-10'88) 


Concluding  Remarks 

Congress  has  displayed  substantial  interest 
in  accelerating  the  dissemination  of  advanced 
automation  technology  to  and  in  United  States 
industry  . 

The  NASA  Systems  Autonomy  Demonstration  Proj¬ 
ect  has  oeen  initiated  in  response  to  this  desire 
cy  Congress  and  will  conduct  Space  Station  automa¬ 
tion  technology  demonstrations  in  1988,  1990, 

1993,  and  '996. 
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Abstract 

\  mimi’er  a  view*.  for  ’he  de'i-i.^pmem  o  problem 

robing  systems  have  emerged  over  the  last  decade  in 
AI  This  paper  will  examine  one  such  view,  the  concept 
of  the  generic  task,  one  of  a  small  set  of  ubiquitous 
cognitive  tasks  that  together  account  for  some  part  of 
human  cognition  In  particular,  we  will  show  how 
hierarchical  classification  is  one  such  generic  task  and 
how  it  is  useful  for  diagnosis  We  will  also  show  that  a 
diagnostic  system  based  on  hierarchical  classification 
naturally  lends  itself  to  solving  a  number  of  issues 
traditionally  associated  with  sensor  calidation 


l.  Introduction 

The  current  generation  of  expert  system  languages - those 

that  are  based  on  rules,  frames,  or  logic - do  not  distinguish 

netween  different  types  of  knowledge  —  based  reasoning  For 
-■xarnpie,  one  would  expect  that  the  task  of  designing  a  car  would 
require  significantly  different  reasoning  strategies  than  the  task 
of  diagnosing  a  malfunction  in  a  car  However,  these 
methodologies  apply  the  same  strategy  (tire  the  rules  whose 
conditions  match,  run  resolution  engine  on  all  propositions  etc  ) 
to  both  design  and  diagnosis,  as  well  as  any  other  task  Because 
of  this,  .t  has  been  argued  that  these  methodologies,  although 
useful,  are  rather  low  level  with  respect  to  modeling  the  needed 
■ask -  level  behavior  In  essence,  these  systems  resemble  an 
assembly  language  for  writing  expert  systems  While  obviously 
iseful.  clearly  approaches  that  more  directly  address  the  higher 
evei  issues  of  knowledge  -  based  reasoning  are  needed  for  the 
next  generation  of  AI  development 

One  example  of  a  higher  level  approach  is  the  generic  tas*(6| 
The  aim  here  is  to  identify  "building  blocks"  of  reasoning  types 
such  that  each  of  the  types  is  both  generic  and  widely  useful  as 
components  of  complex  reasoning  tasks  We  have  identified  to 
date  six  such  generic  strategies,  which  together  account  for  a  very 
arge  portion  of  current  expert  system  capabilities  Each  generic 
task  is  characterized  by 

1  The  kinds  of  information  required  as  input  for  the 
task  and  the  information  produced  as  a  result  of 
performing  the  task 

2  A  way  to  represent  and  organize  the  knowledge  that 
is  needed  to  perform  the  generic  task. 

3  The  process  'algorithm,  control,  problem  solving) 
that  the  task  uses 

As  each  task  and  its  associated  structure  is  identified, 
languages  are  developed  that  encode  both  the  problem  solving 
strategy  and  knowledge  that  is  appropriate  for  solving  problems 
of  that  type  These  languages  facilitate  expert  system 
development  by  giving  the  knowledge  engineer  access  to  tools 
which  work  at  the  level  of  the  problem,  not  the  level  of  the 
machine  Below  is  a  list  of  the  generic  tasks  that  have  already 
been  identified  and  the  tools  that  correspond  to  them: 
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1  Hierarchical  Classification  i*  finding  ’he  categoric* 
in  ,i  classification  hu-ruichv  that  applv  to  the 
situation  being  analyzed  The  tool  lor  classification 
is  CSRL  (4|  (Conceptual  Structures  Representation 
Language)  A  significant  portion  of  expert  systems 
such  as  \1  YCI.N  ( 161  and  PROSPECTOR  (91  can  be 
viewed  as  classification* 

2  Plan  Selection  and  Refinement  is  designing  an  object 
using  hierarchical  planning  DSPLI2I  'Design 
Specialists  and  Plans  Language!  is  the  tool  for  this 
generic  task  The  task  performed  by  the  expert 
system  MOLGEN  !  101  and  Rt  1 121  can  be  viewed  in 
this  way 

3  Knowledge  -  Directed  Information  Passing  is 
determining  the  attribute  of  some  datum  based  on 
the  attributes  of  conceptually  -  related  data  The 
tool  for  this  generic  task  is  IDABLF.  Intelligent 
DAta  Base  LanguagKl  This  task  is  often  used  in 
support  of  other  tasks  such  as  classification  or 
design 

4  Hypothesis  Matching  is  matching  hypotheses  to  a 
situation  using  an  hierarchical  representation  of 
evidence  abstractions  The  tool  for  this  task  is 
HYPER  iHYPothesis  matchER)  Expert  system 
PIP  1 191  can  be  viewed  as  performing  this  task  at 
some  stage  in  its  reasoning 

5  Hypothesis  Assembly  is  constructing  composite 
hypotheses  in  order  to  account  for  some  set  of  data 
PEIRCE  (131  is  the  tool  for  this  task 
INTERNIST  [1 1  and  DENDRAL  (31  systems  largely 
perform  this  task 

As  an  example  of  the  generic  task  concept,  this  paper  will 
address  the  problem  of  diagnosis  and  how  hierarchical 
classification  solves  some  part  of  that  problem  Furthermore,  we 
will  discuss  how  the  problem  of  sensor  validation  naturally 
integrates  into  a  hierarchical  classification  of  diagnosis  In  such 
an  integration,  a  level  of  abstraction  is  provided  that  goes  one 
step  beyond  traditional  approaches  that  rely  strictly  on  hardware 
redundancy 


2.  Classificatory  Problem  Solving 

Diagnosis  as  a  classification  problem  solving  task  is  a  matching 
of  the  data  of  the  problem  against  a  set  of  malfunctions  (i  e 
diseases,  system  failures  etc!  If  the  present  data  is  classified  as  a 
known  malfunction,  then  the  diagnosis  is  completed.  Note  that 
this  is  a  compiled  approach  to  diagnosis  as  it  requires  that  the 
possible  malfunctions  of  the  particular  domain  be  pre  — 
enumerated.  Other  less  well  defined  problems  require  a  deeper 
model  that  relies  on  first  principles  (physic,  chemistry  etc.)  and 


^[n  fact.  Clancey[7]  has  specifically  analyzed  MYCIN  and 
shown  it  to  be  a  kind  of  classification  problem  solving 
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'2.2.  Knowledge  Required  and  its  Organization 

fhe  clarifier  requires  a  pre  -  enumerated  list  of  the  categories 
’rat  t ’  wiil  he  using  Furtnermore.  these  categories  mu?i  be 
■  rgam/ed  .nto  a  hierarchy  in  which  the  ■  hildren  ie  ?he 
'  .inr.odes*  of  a  node  represent  subhypotheses  of  the  parent*  1  e  the 
'-.per.nr  node1  Figure  l  illustrates  a  fragment  of  a  tree  from  a 
hierarchical  classification  system  for  the  diagnosis  ><f  Fuel 
m  •'tern  malfunctions  in  a  car  engine 


Fuel  System  Problems 


Sad  Fuel  Problems  Fuel  Mixture  Problems 


Low  Octane  Water  In  Fuel  Dirt  In  Fuel 


Figure  I;  Fragment  >A' Fuel  System  classification  tree 


Note  that  as  the  hierarchy  is  traversed  from  the  top  down,  the 
categories  or  m  this  particular  case,  hypotheses  about  the  failure 
of  the  fuel  sy^terrii  become  more  specific  Thus  the  children  of  the 
hypothesis  Bad  Fuel  Problems  can  be  broken  into  more  specific 
hypotheses  ofl.ow  Octane.  Water  m  Fuel  and  Dirt  in  Fuel 

Facn  node  in  the  hierarchy  is  responsible  for  calculating  the 
’’degree  of  fit"  or  confidence  value  of  the  hypotheses  that  the  node 
represents  For  example,  the  Bad  Fuel  Problems  node  is 
’•^spons.ibie  for  determining  :f  there  is  a  bad  fuel  problem  and  the 
degree  of  confidence  it  has  in  that  decision  Each  node  can  be 
‘bought  of  as  an  expert  in  determining  if  the  hypothesis  it 
represents  ;s  present  For  this  reason,  each  node  is  termed  a 
'L'eaniist  m  domain  To  create  each  specialist, 

Knowledge  must  be  provided  to  make  this  confidence  value 
leciston  The  general  idea  is  that  each  specialist  specifies  a  list  of 
f  hat  are  important  in  determining  whether  the 
hypothesis  :t  represents  is  present  and  a  fist  o (patterns  that  map 
: umbinations  of  features  to  confidence  values  In  the  Fuel  System 
Problems  -peeia.ist,  such  features  might  include  gas  mileage 
problems,  poor  performance,  difficulty  in  starting  the  engine  etc 
1  Pie  pattern  might  be  that  if  all  the  features  are  present,  then  the 
Fuel  System  Problems  hypothesis  is  likely 


TSpace  and  time  limitations  fimU  'his  paper  to  the  compiled 
wstem  issues,  see  reference  1 141  for  a  detailed  discussion  of  the 
deep  model  issues  and  computational  strategies 


2.3.  The  Control  Strategy  of  Hierarchical 
Classification 

< Jiwn  that  the  Know iccjge  <>f : ho  -■••m  - v. , ■: 
'  pec  1a  lists  in  a  Hierarchy,  now  can  •  n*-  ”.»-r  ir  n.  ••••  •  •*:  •  . 
' r  a  v ersed"*  This  procts>  is  pr  1  mar: I  v  n  » in i>i  1  -hi  <1  * n ro  ,_jr  .1  u*1 

■  I  hypothesis  refinement  called  -’s (<:r>f-n  j.it  .1 

'I'eria’ist  that  establishes  its  hypothe-i'.  v..*-*  »  -.  gn  •  rfi^erv*- 
•  a. i.ie'  •’•'fines  ;lnell  by  activa’mg  ;•  %  ;:»m  .  -  ■{,  — 

'peciaii.its  \  -.peciatist  that  rules  out  -»r  .  1  n  .  p<.- r»--. - 

ha>  a  '>w  confidence  value-  d»e>  riot  -*m<!  it*  .  •  .  ■  . 

'.ii)'Peciaii>l-i,  thus  avoiding  -  nut  •■Ufa*  par!  ■  *r«  •  .. 

t  he  :  *  a -•>n  for  this  becomes  .in  .ini>  a  nen  om.  '  -  .1^...::  1'  n 

-pocialists  are  organized  Him  -*unh\. p*»' m* -c-  •»!  b  .i:  •. 

Problems,  for  example,  are  simply  nioie  detailed  n>  potr.e^es  if 
there  is  no  evidence  for  Fuel  System  Problems  fit  is  ruled  out). 

■  hen  there  is  no  point  in  examining  more  detailed  h*.  pot  he*  is 
■about  failures  of  the  fuel  system 

The  process  of  establish  -  refine  continues  urCti  no  more 
refinements  can  take  place  This  can  occur  either  by  .-‘■jcr.irvg  T.e 
tip  level  hypotheses  of  the  hierarchy  >r  by  ruling  <»at  mid- 
hierarchy  hypotheses 


3.  CSRL,  a  Language  Tool  for  Hierarchical 
Classification  Systems 

CSRL  ‘Conceptual  Structure  Representation  {.ur.guaA'»*  .  *  1 
’anguage  for  writing  hierarchical  ria-sificalion  expert  '■■. -u-m- 
\s  such,  it  allows  a  knowledge  engineer  ’o  do  throe  tr.ir.g- 

1  Create  a  hierarchy  of  malfunction  hypotheses  in  a 
particular  domain 

2  Encode  the  pattern  matching  knowledge  for  each 
hypothesis  into  a  specialist 

3  Control  the  process  of  establish  -  refine  problem 
solving 

3.1.  Encoding  the  Hierarchy  of  Malfunctions 

In  CSRL,  a  hierarchical  classification  system  i,  .implemented  b> 
individually  defining  a  specialist  tor  each  malfunction 
hypothesis  The  super  -  and  sub  -  specialists  of  a  specialist  are 
declared  within  the  definition  Figure  1  is  a  skeleton  of  a 
'peciaiist  definition  for  the  Bad  Fuel  node  from  Figure  1  The 
declare  section  specifies  its  relationships  *.»  other  specialists 
The  other  sections  of  the  specialist  will  be  examined  later 


(Specialist  BadFuei 

(declare  (superspecialist  FuelSystem) 

(subspecial'ists  LowOctane  WaterlnFuel 
Dirt  InFuel ) ) 

(kgs  . . . ) 

(messages  ...)) 

Figure  2:  Skeleton  specialist  for  3a dFue  1 

Designing  a  classification  hierarchy  is  an  important  part  of 
building  a  CSRL  expert  system,  but  the  exact  structure  of  the 
final  system  is  a  pragmatic  decision  rather  than  a  search  for  the 
perfect  hierarchy  The  main  criterion  for  evaluating  a 
classification  hierarchy  is  whether  enough  evidence  is  normally 
available  to  make  confident  decisions  To  decompose  a  specialist 
into  its  subspecialists.  the  simplest  method  is  to  ask  the  domain 
expert  what  subhypotheses  should  be  considered  next  The 
subhypolheses  should  be  subtypes  of  the  specialist’s  hy  pothesis, 
and  will  usually  differ  from  one  another  based  on  a  single 
attribute  (e  g  ,  location,  cause) 


3.2.  Encoding  Pattern  -  Match  Knowledge 

The  knowledge  groups  in  the  kgs  section  contain  knowledge 
•hat  matches  the  features  of  a  specialist  against  the  case  data 
Kacn  knowledge  group  is  used  to  determine  a  confidence  vaiue  lor 
,ome  suoset  of  features  used  by  the  specialist  As  such,  a 
knowledge  group  becomes  an  abstraction  oi  evidence, 
representing  an  evidential  abstraction  of  a  particular  set  0 
...  mires  ;mt»rtant  to  establishing  the  specialist  \  knowledge 
.int'ip  s  implemented  as  a  cluster  of  production  rules  that  maps 
he  values  U  i  list  of  expressions  boolean  and  arithmetic 
per  it.ons  m  data,  values  of  other  knowledge  groups'  '■>  -oijie 
■  ivi-K-in  on  a  discrete  a  mboltc  scale 

\s  an  example.  Figure  S  i»  the  re.evin:  knowledge  group  .a 
fhe  BadFuel  specialist  mentioned  above  It  determines  whether 
•he  svmptoms  of  the  automobile  are  consistent  with  bad  luel 
problems  The  expressions  in  the  match  part  queries  the  user 
who  acts  as  the  database  for  this  case)  concerning  whether  the 
car  is  slow  to  respond,  starts  hard,  has  knocking  or  pinging 
-ounds.  or  has  the  problem  when  accelerating  AskYNU?  is  a  LISP 
function  which  asks  the  user  for  a  Y.  N.  or  U  (unknown)  answer 
from  the  user,  and  translates  the  answer  into  T.  F,  or  L  the 
values  of  CSRL's  three -valued  logic  (Note  that  any  LISP 
•'unction  may  be  used  here)  The  results  of  the  match  expressions 
are  then  compared  to  a  condition  list  in  the  with  part  oi  the 
knowledge  group  For  example,  the  first  pattern  "T  ?  ?"  in  the 
figure  tests  whether  the  first  match  expression  (AskYNU0  "Is 
tr.e  :ar  slow  to  respond")  is  true  ('he  0  means  doesn't 
matter)  If  so,  then  -  T1  becomes  the  value  of  the  knowledge 
group  Otherwise,  subsequent  patterns  T  ?"  or  "?  ?  T"  are 
evaluated  The  value  of  the  knowledge  group  will  be  1  if  no  rule 
matches  This  knowledge  group  encodes  the  following  matching 
knowledge 

If  the  car  is  slow  to  respond  or  if  the  car  starts  hard, 
then  3adFuel  is  not  relevant  in  this  case  Otherwise,  if 
there  are  knocking  or  pinging  sounds  and  if  the  problem 
occurs  while  accelerating,  then  BadFuel  is  highly 
relevant  In  all  other  cases.  BadFuel  is  only  mildly 
relevant 


(relevant  Table 
(match 

(AskYNU?  "13  the  car  slow  to  respond") 
(AskYNU?  "Does  the  car  3tart  hard") 

(And  (AskYNU?  "Do  you  hear  knocking  or 
pinging  sounds") 

(A3kYNU?  "Does  the  problem  occur  while 
accelerating") ) 

with  ( if  T  ?  ? 
then  -3 
el3eif  ?  T  ? 

then  -3 
el3eif  ?  ?  T 
then  3 
el3e  1))) 

Figure  3:  relevant  knowledge  group  of  BadFuel 


Figure  4  is  the  summary  knowledge  group  of  BadFuel  Its 
match  expressions  are  the  values  of  the  relevant  and  gas 
knowledge  group  (the  latter  queries  the  user  about  the  temporal 
relationship  between  the  onset  of  the  problem  and  when  gas  was 
last  bought).  In  this  case,  if  the  vaiue  of  the  relevant  knowledge 


^ln  this  case,  the  values  assigned  are  on  a  discrete  scale  from 
-3  to  ?,  -3  representing  ruled -out  and  3  representing 

confirmed 


gr  >’ip  ,  >  inh  'he  ,j:ue  *:  "'e  g'j  xm-v  _•  rj  -  _•*-  i  •  i  ■  r 

■  he n  ?r  equal  M  ).  men  tne  vaiue  ,i  me  m  we-e.g.- 

group  'ana  consequently  *.ne  confidence  vaiue  a  dxlr  ,e.  '3 

indicating  that  a  bad  fuel  problem  i>  verv  iikeiv 


(suinnary  Table 

(match  relevant  gas 
with  (if  3  ( GE  0) 
then  3 

el  seif  1  (CE  0) 
then  2 

else  if  ?  (LT  0) 

then  -3))) 

Figure  4:  summary  knowledge  group  of  Bad  rue  i 


This  method  of  evidence  combination  allows  the  calculation  of 
the  confidence  value  to  be  hierarchically  organized  That  is.  the 
results  of  any  number  of  knowledge  groups  can  he  further 
abstracted  by  a  knowledge  group  that  can  combine  their  vaiue' 
into  a  single  confidence  value 

3.3.  Encoding  of  Establish  -  Refine  Strategy 

The  messages  section  of  a  specialist  contains  a  list  of  nie'-.ig" 
procedures  which  specify  how  the  specialist  w i i I  respond 
different  messages  from  its  superspeciaiist  Establish  and 
Refine  are  the  predefine  messages  in  CSRL  though  others  may 
be  created  by  the  user  The  establish  message  procedure  of  a 
specialist  determines  the  confidence  value  (i  e  the  degree  of  fit)  of 
the  specialist’s  hypothesis  Figure  5  illustrates  the  establish 
message  procedure  of  the  BadFuel  specialist  relevant  and 
summary  are  names  of  knowledge  groups  of  BadFuel  'see 
previous  section)  self  is  a  keyword  which  refers  to  the  name  of 
the  specialist.  This  procedure  first  tests  the  value  of  the 
relevant  knowledge  group  (If  this  knowledge  group  has  not 
already  been  evaluated,  it  is  automatically  evaluated  at  this 
point  )  If  it  is  greater  than  or  equal  to  0,  then  BadFuei's 
confidence  value  is  set  to  the  value  of  the  summary  knowledge 
group,  else  it  is  set  to  the  value  of  the  relevant  knowledge 
group  A  value  of  +2  or  *3  indicates  that  the  specialist  is 
established.  In  this  case,  the  procedure  corresponds  to  the 
following  strategy 

First  perform  a  preliminary  check  to  make  sure  that 
BadFuel  is  a  relevant  hypothesis  to  hold  If  it  is  not 
(the  relevant  knowledge  group  is  less  than  0).  then  set 
BadFuei's  confidence  value  to  the  degree  of  relevance 
Otherwise,  perform  more  complicated  reasoning  (the 
summary  knowledge  group  combines  the  values  of  other 
knowledge  groups)  to  determine  BadFuei's  confidence 
value 


(Establish  (if  (GE  relevant  0) 

then  (SetConfidence  self  3tu«ary) 
else  (SetConfidence  3elf  relevant))) 

FigureS:  Establish  procedureof  BadFuel 


The  refine  message  procedure  determines  what  subspecialists 
should  be  invoked  and  the  messages  they  are  sent  Figure  6 
shows  a  refine  procedure  which  is  a  simplified  version  of  the  one 
that  BadFuel  uses  subspecialists  is  a  keyword  which  refers 
to  the  subspecialists  of  the  current  specialist  The.  procedure  calls 
each  subspecialist  with  an  establish  message  If  the 
subspecialist  establishes  itself  <♦?  tests  if  the  confidence  value  is 
+  2  or  -t-3),  then  it  is  sent  a  ref  ine  message 


(Refine  (for  specialist  in  subspedal  ists 

do  (Call  specialist  with  Establish) 
(if  (4-?  spec  ial  ist ) 
then  (Call  specialist 
with  Refine)))) 

Figure  b:  Evamou*  n-f'np  orocedure 


The  Computational  Ad  vantages  of 
Hierarchical  Classification 

Inc  :’i it i* > r  .ul  v  .rv  ..g*-  •»!  t  hierarchical  clarification  <v*tem  1- 
•  ne  organization  of  both  the  hierarchy  of  malfunctions  and  the 
knowledge  groups  within  a  specialist  This  organization  allows  an 
efficient  examination  of  the  knowledge  of  the  system  based  on 
need 

Consider  again  the  hierarchy  of  Figure  i  The  problem  solving 
begins  by  evaluation  of  the  specialist  Fuel  System  Problems  if 
that  specialist  establishes,  then  the  two  sub  — specialists  Bad  Fuel 
Problems  and  Fuel  Mixture  Problems  are  invoked  If  however,  the 
Bad  Fuel  Specialist  does  not  establish.  *hen  none  of  its  *ub  — 
specialists  will  ne  invoked  Thus,  if  a  specialist  rules  out  i  e  does 
not  establish),  then  none  of  the  knowledge  of  the  sub  — specialists 
need  be  .  on 

The  >ame  i-  true  of  the  knowledge  groups  in  the  specialist  '  )nl> 
’hat  Knowledge  necessary  to  confirm  or  deny  the  knowledge  group 
is  run  If  a  row  of  the  knowledge  group  matches,  then  none  of  the 
subsequent  rows  are  evaluated  Again,  this  results  in  running 
>>nly  the  knowledge  necessary  for  the  problem  at  hand 

Compare  this  with  other  so -called  hierarchical  approaches  to 
diagnosis  The  fault  tree  is  a  sequence  of  causally  related  events 
that  leads  to  an  observable  symptom  in  the  system  Given  an 
initial  malfunction,  all  possible  causal  results  of  the  event  are 
traced  out.  terminating  with  the  symptoms  that  would  be 
observed  by  a  human  diagnostician  When  applied  to  an  entire 
system,  the  result  is  a  network  of  events  that  represent  all  the 
causal  relationships  of  the  system’s  constituent  parts  While 
useful  in  design  tasks,  application  of  fault  trees  to  diagnosis  has  a 
number  of  problems 

1  The  combinatorial  fan-out  from  an  initial  event 
can  be  very  large  This  makes  the  job  of  creating  and 
traversing  the  network  difficult  Compare  this  with 
the  abstraction  of  hypotheses  in  hierarchical 

•  lassification  systems  Each  node  in  the  hierarchy 
represents  a  malfunction  hypothesis  that  is  listed  in 
more  detail  through  its  sub  —  specialists  If  many 
->ub  -  specialists  occur  in  the  hierarchical 
decomposition  of  the  domain,  more  levels  of 
abstraction  can  be  introduced  to  iimit  the  fan— out 
Such  abstraction  does  not  exist  in  fault  tree 
representations 

2  Fault  trees  make  no  attempt  to  limit  the  number  of 
nodes  of  the  network  that  must  be  evaluated  Given 
a  significant  event,  ai!  possibilities  are  examined 
However,  hierarchical  classifiers  make  use  of  the 
abstraction  of  malfunction  hypotheses  to  limit  the* 
number  of  nodes  that  must  be  examined  based  on  the 
data  of  the  case 


5.  Practical  Applications  of  Hierarchical 
Classification 

A  number  of  diagnostic  systems  have  been  built  using  the 
hierarchical  classification  approach  provided  by  the  CSRL  tool 
This  section  enumerates  some  the  these  applications  and  their 
domains 


It  should  be  noted  that  of  the  following  \  M.*cn  * 

strictly  a  pedagogical  system,  the  Nuesear  P  *w«*r  •  rm  <'h*"n.ca; 
Engineering  systems  are  initial  exoiorations  ■  >r  •.* 

developed  systems  and  Red.  WF.LDEX  and  ROM  M)  ire 
developed  to  be  used  in  real  world  situation* 

Auto  -  Mech  (20| 

Auto  -  Mech  is  an  expert  system  which  diagnoses  fuel  problem* 
m  automobile  engines  The  purpose  of  the  fuel  -vstem  s  *o 
deliver  a  mixture  of  fuel  and  air  to  the  air  r\  linders  of  t  he  engine 
It  can  he  divided  into  major  ^uhsv*  terns  ifuei  delivery.  air  :n?jKe. 
.arbuertor,  vacuum  manifold)  which  <-orn  -pond  •<»  ,  1 . .  1 . 

hv  potheses  about  fuel  sv  >tem  fault* 

Auto— Mech  consists  of  34  CSRL  specialists  in  a  hierareny 
which  vanes  from  four  to  six  levels  deep  Before  running.  Auto- 
Mech  collects  some  initial  data  from  the  user  This  includes  the 
major  symptom  that  the  user  notices  tsuch  as  stalling)  and  the 
situation  when  this  occurs  ( e  g  ,  accelerating  and  cold  engine 
temperature).  Any  additional  questions  are  asked  while  Auto- 
Mech’s  specialists  are  running  The  diagnosis  continues  until  tne 
user  is  satisfied  that  the  diagnosis  is  complete 

A  major  part  of  Auto  -  Mech's  development  was  determining  the 
assumptions  that  would  be  made  about  the  design  of  tne 
automobile  engine  and  the  data  that  the  program  would  use 
Different  automobile  engine  designs  have  a  significant  effect  .n 
the  hypotheses  that  are  considered  A  carbureted  engine,  for 
example,  will  have  a  different  set  of  problems  than  a  fuel  injected 
engine  (the  former  can  have  a  broken  carburetor!  The  data  wj* 
assumed  to  come  from  commoniv  available  resources  The  variety 
of  computer  analysis  information  that  is  available  to  mechanics 
today  was  not  considered  in  order  to  simplify  building  Auto- 
Mech 

Red  118| 

Red  is  an  expert  system  whose  domain  is  red  blood  cell  antibody- 
identification  An  everyday  problem  that  a  blood  bank  contends 
with  is  the  selection  of  units  of  b/ood  for  transfusion  during  major 
surgery  The  primary  difficulty  is  that  antibodies  in  the  patient’s 
blood  may  attack  the  transfused  blood,  rendering  tne  new  blood 
useless  as  well  as  presenting  additional  danger  to  the  patient 
Thus  identifying  the  patient's  antibodies  and  selecting  blood 
which  will  not  react  with  them  is  a  critical  task  for  nearly  all  rrd 
blood  transfusions 

The  Red  expert  system  is  composed  of  three  major  subsystem*, 
one  of  which  is  implemented  in  CSRL  The  non -CSRL 
subsystems  are  a  data  base  which  maintains  and  answers 
questions  about  reaction  records  1 reactions  of  the  patient's  blood 
in  selected  blood  samples  under  a  variety  of  conditions),  and  a 
overview  system,  which  assembles  a  composite  hypothesis  of  the 
antibodies  that  would  best  explain  the  reaction  record.  CSRL  is 
used  to  implement  specialists  corresponding  to  the  common  blood 
antibodies  and  to  each  antibody  subtype  (different  ways  ’hat  the 
antibody  can  react) 

The  major  function  of  the  specialists  is  to  rule  out  antibodies 
and  their  subtypes  whenever  possible,  thus  simplifying  the  job  of 
the  overview  subsystem,  and  to  assign  confidence  values, 
informing  overview  of  which  antibodies  appear  to  be  more 
plausible  The  specialists  query  the  data  base  for  information 
about  the  lab  test  results  and  other  patient  information,  and  also 
tell  the  data  base  to  perform  certain  operations  on  reaction 
records 

Complex  Mechanical  Systems 

CSRL  has  been  used  in  creating  expert  systems  that  do 
diagnosis  of  faults  both  in  the  domain  of  Nuclear  Power  Plants 
and  in  the  domain  of  Chemical  Engineering 

The  Nuclear  Power  Industry  must  be  very  capful  m  the 


A 


maintenance  of  running  power  piants  s.nee  mistakes  can  prove 
costly  not  only  in  terms  of  power  plant  damage  hut  also  m  terms  of 
radiation  leakage  and  broad  environmental  damage  Nuclear 
Power  Plants  are  therefore  heavily  monitored  in  manv  areas,  so 
heavily  in  fact  that  .t  is  difficult  if  not  . mpo^siblei  for  »he 
operator  to  maintain  an  understanding  of  just  what  exactly  is 
going  on  rhe  N’ucieai  Power  Plant  expert  ^v^tem  i  11 !  is  designed 
'.jke  in  large  amounts  of  data  and  cla^ifi  tr.rtm  into  one  «>f 
lupi  uximati*!  v  J ^  different  failurn>  <  >ne  advantage  of  the  CSRL 
ppi'Kicn  mat  the  operator  can  he  intortnedoi  a  nigh  level  view 
■  •  he  pmhlem  *f  no  -i pec i fie  :afiur**  u:i  n»-  d.  -c-*v  t-ri-d 

f’he  pr->hiem>  .»i  *h*-  Chemical  Engineering  Plant  are  amil.ic 
hut  it  does  have  a  number  d  differences  While  safety  i»  also  of 
concern,  there  is  also  the  problem  of  product  quality  in  a 
Chemical  Engineering  Plant.  If  a  malfunction  occurs  that 
produces  an  unusable  product,  trie  operation  must  be  brought 
quickly  hack  into  line  or  large  amounts  of  material  will  be  wasted 
The  Chemical  Engineering  expert  system  ( 17|  does  diagnosis  of  a 
typical  reactor  producing  a  solid  product  as  a  result  of  the 
reaction  of  liquid  product  and  oxygen  It  consists  of  approximately 
50  specialists  that  represent  hypotheses  about  failures  of  the 
various  physical  parts  of  the  plant  In  addition  to  data  that 
monitors  the  state  of  the  reactor,  these  specialists  also  use  data 
about  product  quality  to  make  the  confidence  value  decision 

Other  Real  World  Uses  of  CSRL 

CSRL  is  being  used  to  develop  two  commercial  systems  b>  the 
Knowledge  Bused  Systems  group  at  the  Battelle  Columbus 
Institute  WELDEX  and  ROMAD  are  diagnostic  systems  for. 
respectively,  detecting  welding  defects  and  evaluating 
machinery  A  brief  description  of  WELDEX  follows 

WELDEX  identifies  possible  defects  in  a  weld  from 
radiograpmc  data  of  the  weld  Industry  standards  and 
regulations  require  careful  inspection  of  the  entire  weld  and  a 
very  high  level  of  quality  control.  Thus  for  industries  which  rely 
on  welding  technology,  such  as  the  gas  pipeline  industry, 
radiograph  inspection  is  a  tedious,  time-consuming,  and 
expensive  part  of  their  operations 

This  problem  can  be  decomposed  into  two  tasks  visual 
processing  ol  the  radiograph  to  extract  relevant  features  of  the 
weld,  and  mapping  'hese  visual  features  to  the  welding  defects 
wnich  give  rise  to  them  WELDEX  is  intended  to  perform  the 
•second  task  The  current  prototype  consists  of  25  CSRL 
-peciansts  that  are  organized  around  different  regions  of  the 
w  eid.  taking  advantage  of  the  fact  that  each  class  of  defects  tends 
to  occur  in  a  particular  region  The  knowledge  groups  in  these 
specialists  concentrate  on  optical  contrast,  shape,  size  and 
location  of  the  radiograph  features.  A  customer  version  of 
WELDEX  is  currently  being  developed  Future  work  is 
anticipated  on  developing  a  visual  processing  system  whose 
output  would  be  processed  by  WELDEX,  thus  automating  both 
parts  of  the  radiograph  inspection  problem 


6.  Data  Validation  in  Hierarchical  Classification 

We  have  examined  in  some  detail  how  hierarchical 
classification  lends  itself  to  performing  diagnosis.  However,  there 
are  a  number  of  issues  that  exist  m  real-world  systems  that 
must  yet  be  addressed  For  example,  most  of  the  work  in  A!  in 
diagnosis  assumes  the  observations  given  to  an  expert  system  are 
reliable  In  fact,  it  has  always  been  understood  that  in  real- 
world  situations  data  can  be  unreliable  and  that  human  experts 
can  still  reach  reliable  diagnostic  conclusions  despite  this 
unreliable  data 

As  an  example,  examine  the  domain  of  complex  mechanical 
systems  like  Nuclear  Power  Piants  or  Chemical  Processing 
Plants  Even  before  the  advent  of  AI,  system  monitoring  aids 
were  designed  that  provided  the  operator  with  automatically 


validated  data  These  designs  centered  m  momtunng  each 
important  system  datum  'pressure,  temperature  etc  j  *>ih  a 
number  of  hardware  sensors.  This  muitiplicitv  of  -ensors 
provided  a  redundancy  of  information  from  which  a  more  reliable 
value  was  extracted  Based  on  this  hardware  redundance,  a 
number  ol  techniques  were  developed  to  validate  a  datum  s  value 

1  Providing  more  than  one  sensor  of  the  same  kind  to 

monitor  some  datum  Loss  of  one  sensor  therefore 
does  not  preclude  data  gathering  and  any 
disagreements  among  the  -»ensors  can  be  resolved 
statistically  for  the  overall  datum  value  for 
example,  to  measure  the  temperature  •»!  a  eh<  micjl 
leaction.  multiple  temperature  ■••aid 

used  in  the  reactor  and  their  statistical  average* 
given  as  the  overall  temperature  value 

2  Providing  different  kinds  of  sensors  to  monitor  a 
datum.  This  situation  provides  the  same  redundancy 
as  1)  as  well  as  minimizing  the  possibility  of  some 
kinds  of  common  fault  problems  That  is.  certain 
events  that  inactivate  a  particular  sensor  type  may 
not  affect  sensors  of  a  different  type  Continuing 
with  the  example  of  1)  above,  half  of  the  sensors 
might  be  thermocouples  while  the  other  half  might 
be  mechanical  temperature  sensors 

3  L’sing  sensors  in  several  different  locations  to  infer 
a  datum  value.  In  this  situation,  datum  values  are 
monitored  not  only  directly  but  also  inferred  from 
other  system  data  based  on  well-established 
relationships  For  example,  while  the  temperature 
of  a  closed  vessel  may  be  directly  monitored,  it  can 
be  inferred  from  the  measurement  of  the  pressure 
using  the  PV  -  nrT  equation 

While  hardware  redundancy  allows  some  data  validation,  it  has 
a  number  of  drawbacks. 

1  The  expense  of  installing  and  maintaining  multiple 
sensors  for  each  important  datum  greatly  increases 
the  cost  of  the  system 

2  Common  fault  failures  still  happen,  especially  as  the 
result  of  severe  operating  failures 

3  Human  operators  and  engineers  resolve  many  such 
diagnostic  problems  using  conflicting  and  even 
absent  data  In  other  words,  human  experts  are 
more  tolerant  of  bad  data  whether  it  has  been 
validated  or  not. 

The  following  simple  example  will  help  in  examining  point  3) 
and  other  ideas'*.  Consider  the  mechanical  system  diagrammed 
in  Figure  7  with  data  values  indicated  in  Figure  8  It  is  a  closed 
vessel  with  two  subsystems,  a  cooling  system  and  a  pressure  relief 
system.  The  vessel  is  a  reactor  which  contains  some  process 
(nuclear  fission,  chemical  reactions  etc  )  that  produces  both  heat 
and  pressure.  The  data  values  of  Figure  8  indicate  that  the 
temperature  of  the  reactor  vessel,  as  determined  by  sensor 
hardware,  is  above  acceptable  limits  Assume  that  there  are  two 
possible  causes  of  the  high  reactor  temperature:  either  the 
cooling  system  has  failed  or  the  pressure  relief  system  has  failed 
and  the  added  heat  has  overpowered  the  functioning  cooling 
system.  Given  the  sensor  readings,  what  would  be  the  diagnostic 
conclusion?  The  data  conflict  is  the  norma/  pressure  and  cooling 
system  readings  and  the  abnormal  pressure  relief  system 
readings.  The  failure  of  the  pressure  relief  system  is  plausible, 
data  indicates  its  failure  and  no  other  system  failure,  but  such  a 
failure  expects  the  pressure  to  be  high?  The  step  to  take  is  to 
assume  both  the  pressure  relief  system  to  have  failed  and  the 
pressure  sensor  to  be  incorrect. 

^Note  that  the  ideas  presented  here  have  been  used  on  more 
complicated  real-world  systems (8,  17,  151  This  example  has 
been  condensed  from  them  for  expository  clarity 


The  process  shown  above  demonstrates  data  validation  at  a 
higher  level  than  that  of  simple  sensor  hardware  validation  In 
the  example,  the  pressure  system  has  failed  despite  the  lack  of  a 
hi^h  pressure  datum  However,  there  is  other  strong  evidence^ 
that  the  pressure  system  has  indeed  failed  The  human  reasoner 
expects  the  pressure  datum  to  be  high  since  the  preponderance  of 
other  data  indicate  a  malfunction  That  is,  the  human  reasoner  in 
pursuing  likelv  diagnostic  conclusions  discovers  a  plausible 
diagnostic  conclusion  that  meets  all  but  i  in  this  casei  one 
expectation  The  important  points  to  note  are  that 

l  A  diagnostic  conclusion  can  and  -hi mid  be  made 
ha.^ed  on  the  preponderance  of  other  ev  idencc 

The  datum  value  *hat  does  not  meet  expectation 
should  be  questioned  and  further  investigation  of  its 
true  value  made 

This  is  not  to  say  that  hardware  redundancy  is  not  useful  for 
solving  problems  of  conflicting  sensor  reports  The  point  is  that 
hardware  redundancy  is  made  more  useful  in  service  of  the  higher 
level  of  redundancy  provided  by  diagnostic  expectations. 

Also  note  that  this  process  involves  redundancy,  not  at  the  level 
of  sensor  hardware,  but  at  the  level  of  diagnostic  expectation  This 
is  a  redundancy  of  information  that  allows  questioning  'and 
subsequent  validation)  of  data  based  on  multiple  expectations  of 
diagnostic  conclusions  If  a  conclusion  is  Tkely,  but  not  all  of  its 
expectations  are  met,  then  those  now  questionable  values  are 
investigated  by  more  computationally  expensive  techniques 
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Figure  7:  An  Example  Mechanical  System 


While  space  limitations  restrict  the  scope  of  our  discussion,  the 
following  section  present  the  basic  ideas  concerning  how  the 
expectations  of  data  can  be  naturally  encoded  in  a  hierarchical 
classification  system  For  further  detail,  please  see  (51. 


6.1.  Establishing  a  Questionable  Datum 

The  process  of  establishing  a  questionable  datum  involves  two 
steps  First,  establish  some  expectations,  using  local  knowledge 
or  the  context  of  other  nodes  Second,  use  those  expectations  to 
flag  some  particular  data  value  as  questionable 

The  expectations  of  a  malfunction  are  embodied  in  the 
knowledge  group  The  knowledge  group  mechanism  was  designed 
to  give  a  rating  of  pattern  fit  to  data  If  the  fit  is  not  perfect,  those 
data  values  not  meeting  expectations  are  identified  as 
questionable  In  the  example  of  Pressure  Relief  Valve  Failure, 
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Figure  8:  Sensor  Values  for  the  Example 


evidence  exists  that  the  valve  has  failed  even  though  the  pressure 
is  normal  The  lack  of  perfect  fit  between  data  and  pattern  aitow 
the  pressure  value  to  be  identified  as  questionable  Diagnosis 
continues  despite  apparent  data  conflict  since  enough  evidence 
exists  for  establishing  the  malfunction  hypothesis 

In  establishing  all  the  ancestors  of  any  one  node,  a  context  of 
problem  solving  is  created  which  provides  a  set  of  implicit 
expectations  Examine  the  example  hierarchy  of  Figure  7  In 
order  to  establish  the  malfunction  hypothesis  Relief  Value 
Failure,  the  malfunction  hypothesis  Pressure  System  Failure 
must  have  established.  In  the  context  of  considering  the  Valve 
Failure,  one  can  assume  that  some  expectations  were  created 
based  on  the  establishment  of  the  Pressure  System  Failure  node 
and  other  ancestors  Since  these  expectations  always  exist  when 
considering  Valve  Failure  i.e,  you  can’t  get  to  Valve  Failure 
without  establishing  Pressure  System  Failure,  they  can  be  hard  - 
coded  into  the  Valve  Failure  Node 

How  expectations  are  used  for  the  Pressure  Relief  System 
Failure  node  is  shown  in  Figure  9  A  modification  is  made  to  the 
standard  knowledge  group  first  seen  in  Figures  3  and  4  that 
allows  the  expert  to  indicate  both  a  match  value  for  the  group  and 
a  set  of  data  that  do  not  meet  the  expectations  established  at  this 
stage  of  the  problem  solving  This  is  done  by  adding  the  keyword 
Data-To-Question  to  any  pattern  of  the  knowledge  group, 
followed  by  a  list  of  those  data  values  that  are  considered 
questionable  by  the  expert.  Thus,  Pressure  Relief  Failure 
establishes  (based  on  other  data  features)  despite  the  lack  of  a 
change  of  pressui  e  However,  in  establishing  the  node,  one  should 
question  why  the  pressure  did  not  change  as  expected  This  is 
done  by  associating  with  the  first  pattern  (the  one  that  matches 
when  the  knowledge  group  establishes)  the  pressure  datum  by  use 
of  the  Data-To-Question  keyword  If  that  pattern  is  matched, 
then  the  match  value  is  returned  but  the  indicated  data  value  is 
placed  in  a  list  of  questionable  data  values  will  be  later  in  detail 
If  the  match  value  is  high  enough,  the  node  establishes  itself 
despite  the  existence  of  conflicting  data  That  is,  if  there  is 
enough  evidence  to  show  a  malfunction  despite  a  conflicting 
value,  the  problem  solving  may  continue  However,  it  may  be  the 
case  that  the  value  being  questioned  is  of  vital  importance  to 
establishing  the  node  The  match  value  will  reflect  this,  the  node 
will  not  establish,  but  the  data  will  still  be  placed  on  the 
questionable  data  list  After  an  initial  run  of  the  problem  solver, 
the  questionable  data  list  will  consist  of  datum  values  that  did  not 
meet  the  expectations  of  some  node. 


'’In  the  example,  this  evidence  is  that  there  is  a  failure  of  the 
relief  valve  system  which  is  part  of  the  pressure  system 


(pressure  Table 
( match 

(AskYNlH  "'s  the  Pressure  Alarm  on'7") 
(AskYNlP  "Is  the  Condenser  Pressure  High?") 
(AskYNU?  "Is  the  Temperature  Alarm 
act  i  vated?" ) 

with  (if  F  T  T  Data- To  Question  (pressure) 
'sen  ' 
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7.  Conclusion 

Hierarchical  cij'SHVation  s  an  approach  to  doing  diagnosis  for 
-yv -items  with  complied  ^now.edge  about  the  types  ot  malfunctions 
‘hat  o<cir  It  ->rgani/.e>  ’h;>  knowledge  hierarchically  and 
pn-v.des  ind  efficient  ..ontroi  strategy  'or  examining  those 
malfunction  nypo'hese^  na^ed  on  *he  data  of  the  problem  at  nund 
Furthermore.  fhe  process  of  -ensor  validation  nas  been  shown  to 
be  a  natural  part  •  f  hierarchical  classification  By  adding  a 
simple  change  to  tne  knowledge  group.  data  values  that  do  not 
meet  expectations  car.  be  identified  and  more  powerful 
computational  methods  applied  to  resolve  the  conflict 
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A BS  T  R  ACT 

*  :v',  edge  baaed  systems  (KBS)  Save 

not  ye:  met  expectations.  Situations 

■  rat  haven't  been  anticipated  or  planned 
for  -use  be  bandied  by  KBSs  if  they  are 
to  be  useful  Cor  complex  space  support 
applications.  This  requirement  implies 
that  "deep  Knowledge"  about  the  domain 
must  be  modeled  so  the  KBS  can  "reason" 
about  the  situation  and  generate  and 
implement  a  plan  for  recovery  or  action. 
Domains  may  be  represented  using  several 
schemes,  each  having  limitations  that 
prevent  them  from  meeting  expectations 
for  KBS's.  Ford  Aerospace  has  developed 
a  representation  approach  that  is  a 
hybrid  of  other  schemes.  Because  this 
approach  represents  Knowledge  in  a 
complete,  consistent  and  unambiguous 
manner,  generic  approaches  to  processing 
domain  information  can  be  developed,  and 
unplanned  situations  can  be  analyzed  and 
"reasoned"  about.  This  approach  promises 
to  meet  most,  and  possibly  all,  of  the 
space  support  requirements  for  KBSs. 
Ford  Aerospace's  KBS  architecture  has 
been  applied  to  several  space  support 
activities  in  an  effort  to  test  its  full 
capabilities  in  being  able  to  "reason 
about"  and  "analyze"  new  situations. 
Satellite  autonomy,  equi  pmen t  maintenance 
and  networK  control  applications  are 
among  those  discussed. 


I .  Space  Support 
Application  Requirements 

Knowledge  Based  Systems  (KBS)  have 
been  successfully  applied  to  automate 
some  space  support  functions,  but  the 
expectations  have  usually  exceeded  the 
implemented  capability.  In  some  areas  of 
satellite  control  it  is  acKnowledged  that 
no  existing  KBS  could  replace  the  expert 
facing  an  unforeseen  event.  For  example, 
a  solar  storm  caused  a  bit  change  in  the 
solar  array  adapter  control  of  a  satel¬ 
lite  wmch  locKed  the  solar  array  tracK- 
mg  mechanism.  (see  Figure  1.)  As  the 
satellite  progressed  along  its  orbit,  the 
solar  energy  impinging  on  the  panels 
decreased  because  the  panel  could  not 
maintain  its  normal  perpendicular  posi¬ 
tion  to  the  sun's  cays.  Eventually  the 
power  from  the  soiar  array  was  so  low 


r  ha  t  the  vehicle  automatically  shed  its 
powpr  load  and  turned  off  the  attitude 
control  subsystem.  The  fumbling  vehicle 
was  detected  and  placed  :n  a  safe,  sun 
pointing,  spun.up  configuration.  Analysis 
of  telemetry  data  showed  the  cause  of  the 
locKed  solar  array  to  be  a  spurious  bit 
change.  This  bit  change  was  corrected 
and  the  procedure  for  returning  to  a 
stable  earth  pointing  attitude  was 
started.  The  approved  procedure,  when 
implemented.  failed  to  result  :n  'he 
expected  earth  pointing  attitude. 
Factory  experts  were  consulted  and  it  was 
decided  that  the  thrusters  ..ad  in  the 
maneuver  impinged  on  the  solar  array 

panels,  imparting  forces  that  prevented 
normal  earth  acquisition.  A  second  earth 
acquisition  maneuver  was  planned  using  a 
different  set  of  thrusters.  Known  to  be 

directed  away  from  the  solar  panel  as 
oriented.  This  procedure  was  a  success 
and  the  satellite  was  returned  to  its 
functional  earth  pointing  attitude. 

Another  example  involves  an  R&D 
satellite  which  lost  earth  locK  shortly 
after  initial  earth  pointing  attitude 
control  had  been  established.  There  was 
no  apparent  reason  (e.g.,  sun/moon  inter¬ 
ference,  eclipse,  power  decrease.  etc.) 
for  the  problem.  A  close  looK  at  the 
telemetry  data,  just  prior  to  start  of 

vehicle  tumbling,  showed  that  the  earth 
sensor  data  being  used  by  the  on  board 

software  was  not  the  expected  data. 
Further  research  revealed  that  che 
installed  earth  sensor  design  character¬ 
istic"  were  different  from  those  written 
in  the  control  software.  A  corrected 
version  of  the  control  software  was 
uploaded  to  the  satellite.  The  earth 
acquisition  procedure  was  repeated  and 
the  attitude  control  system  maintained 
earthlocK.  While  we  may  not  expect  KBSs 
to  automat ica 1 ly  correct  the  on-board 
software  we  do  expect  KBSs  to  diagnose 
the  problem  immediately  upon  receipt  of 
the  first  unexpected  earth  sensor  data 
and  recommend  a  solution  to  the 
operator.  This  requirement  implies  that 
orbital  (flight)  information  must  be 
available  and  "deep  Knowledge"  about  the 
satellite  must  be  modeled  so  that  the  KBS 
can  "reason"  about  the  situation  and 
generate  and  implement  a  plan  for 
recovery  or  action. 
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II.  Alternative  KBS 
Appccdcr.es 

Domains  may  be  represented  using 
rules,  frames,  networks  or  other 
schemes .  Some  of  the  more  common  repre¬ 
sentation  alternatives  are  compared  on 
Table  1.  They  all  fall  short  of  meeting 
the  expectations  listed  in  Figure  2  for 
k'BSs  in  space  support  applications.  For 
example,  generic  processing  is  provided 
by  the  inference  engire  in  rule- based 
systems,  but  this  processing  is  limited 
to  pattern  matching  between  situational 
data  and  data  referenced  within  rules. 
To  allow  for  more  powerful  generic 
processing,  the  knowledge  must  have  a 
better  defined  structure  than  that  cur¬ 
rently  used  for  rules.  The  structure 
itself  includes  information  that  can  then 
be  used  by  processing  modules  to  perform 
more  specific  functions  than  pattern 
matching.  In  addition,  the  flexibility 
(oc  lack  of  structure)  of  rule- based 
systems,  makes  system  Verification  and 
Validation  (V&V)  nearly  impossible 
because  it  is  difficult  to  isolate  a  part 
of  the  system  for  test:  Any  rule  that 

fires  may  generate  unexpected  side 
effects  within  other  parts  of  the  system. 

For  a  KBS  to  reason  about  or  adapt  to 
events  or  situations  for  which  if  was  not 
preprogrammed,  deep  or  causal  knowledge 
about  the  domain  must  be  represented  in 
an  explicit  manner  that  can  be  accessed 


by  the  reasoning  modules.  K .  w  •? 
represented  within  rules  or  methods  :s 
only  used  during  actual  execution.  and 
only  the  result  of  the  execution  is 
available  for  reasoning;  the  information 
contained  within  rules  or  methods  is 
inaccessible  to  a  problem  solving  module. 

Major  inadequacies  of  all  existing 
systems  meant  to  be  used  for  real  space 
support  applications  are  presented  on 
Figure  2  and  include:  the  need  for  AI 

expertise  during  system  development,  the 
inability  to  handle  situations  which 
haven't  been  explicitly  entered  into  the 
knowledge  base,  the  lack  of  a  formal  V&V 
approach,  and  the  inability  to  provide 
real-time  solutions  to  a  complex  situ 
a  t ion . 

Ford  Aerospace  has  developed  a  KBS 
approach,  called  PARAGON,  that  is  a 
hybrid  of  other  approaches.  It  promises 
to  meet  most,  if  not  all,  desired 
cha r ac t er  i s t  i cs  of  a  space  support  krs 
The  domain  is  modeled  using  a  semantic 
network  approach  where  the  concepts 
(e.g.,  physical  or  non-physical  objects), 
appropriate  characteristics  of  the  con 
cepts  (e.g.,  height,  weight,  color), 
interaction  or  relationships  with  ocher 
domain  concepts  (e.g.,  electrically  con¬ 
nected,  heat  supplied  by,  etc.),  and  the 
behavior  of  the  concept  (e.g.,  states  it 
may  be  in  such  as  on,  off,  or  idle:  what 
events  occur  while  in  each  state: 


Figure  1.  Example  -  Satellite  Anomaly 
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Table  1.  Comparison  of  Representation  Alternatives 

REPRESENTATION  DESCRIPTION  STRENGTHS  WEAKNESSES 
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Figure  2.  Knowledge  Rased  Systems  -  The  Expectation  Gap 
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and  what  causes  transition  from  one  state 
to  ano  c  he  r )  are  described  via  a  graphic 
interface  and  menus.  Typing  can  be 
limited  to  assigning  names  to  concepts, 
states,  etc. 

As  'he  model  is  being  developed, 
PARAGON  collects  the  information  and 
automatically  translates  it  to  a  machine 
representation  useful  for  inferencing  and 
people-  solving.  Character ist ics , 

relationships  and  behavior  must  be 
defined  precisely  and  within  constraints 
to  avoid  the  problem  of  ambiguous  defini¬ 
tion  found  m  most  semantic  networks  and 
to  allow  formal  V&V.  The  approach  allows 
PARAGON  to  au t oma t ica 1 ly  generate  LISP 
code  so  a  simulator  for  the  domain  is 
available  to  test  behavior  and  display 
characteristics  for  verification  by  the 
system  developer.  Some  minor  mod  i  f  i 
cations  would  allow  generation  of  other 
types  of  code  such  as  C.  PASCAL.  FORTRAN, 
etc.,  so  the  resultant  model  of  the  know¬ 
ledge  base  could  be  used  as  a  stand-alone 
simulator  tunning  on  a  non-LISP  machine. 

During  development  of  this  portion  of 
PARAGON.  two  guiding  principles  were 
followed:  that  the  model  be  declarative 

and  consistent.  We  wanted  the  model  to 
be  declarative  so  all  control  knowledge 
(or  algorithms  tor  processing  knowledge) 
would  be  eliminated  from  the  knowledge 
base  and  all  knowledge  describing  each 
concept  would  be  internal  to  the  data 
structure  itself.  Semantic  networks 
allow  representation  of  a  domain  with  no 
conrr-i  or  processing  of  knowledge 
required.  Also  the  representation 

elements  are  easy  for  engineers  to  work 
with.  Therefore  this  was  our  starting 
point.  However,  any  type  of  relationship 
is  allowed  in  a  "pure"  semantic  network 
which  can  lead  to  inconsistencies. 
Therefore,  constraints  had  to  be  imposed 
on  the  way  concepts  and  relationships  are 
defined  and  the  way  relationships  inter¬ 
act  to  ensure  consistency.  Other  aspects 
of  representation  schemes  such  as  frames, 
rules  and  blackboard  were  used  to  con¬ 
strain  the  knowledge  base  representa¬ 
tion.  Since  many  different  systems  and 
users  may  want  to  access  the  same  know¬ 
ledge  base  for  a  variety  of  purposes  and 
since  two  or  more  knowledge  bases  may 
have  to  interact  to  find  a  solution  for  a 
user.  consistency  was  absolutely  an 
essential  requirement  for  space  support 
appi  icat  ions . 

Since  the  domain  model  in  PARAGON 
contains  no  control  or  processing  know¬ 
ledge,  reasoning  modules  have  been  and 
are  being  developed  to  perform  their 
function  on  the  domain  model,  independent 
of  specific  domain  information.  Cogni¬ 
tive  psychologists.  Artificial  Intelli¬ 
gence  (AI)  researchers  and  knowledge 


engineers  have  identified  the  following 
most  commonly  used  processes  or 
strategies: 

1.  Monitoring  (data  interpret  a 

tion):  comparing  observations  to  planned 

(desired)  situation 

2.  Interpretation  (situation  assess 

sent):  inferring  situation  description 

from  sensor  data 

3.  Diagnosis:  inferring  cause  of 

system  malfunctions  from  observables 

4.  Debugging:  prescribing  remedies 

for  malfunctions 

5.  Prediction:  inferring  likely 

consequences  of  given  situat  ions/act  ions 

6 .  Design:  configuring  or  modifying 
objects  uuuei  constraints,  such  as  avail¬ 
able  resources 

7.  Goal  Determination:  debugging, 

prediction,  design  and  resource  alloca 

cion  (knowing  desired  situation) 

8.  Planning:  designing  actions  to 

achieve  goal 

9.  Repair  (corrective  action): 
executing  a  plan  to  administer  a  pre¬ 
scribed  remedy 

10.  Instruction:  diagnosing, 

debugging  and  repairing  student  behavior 

The  reasoning  process  followed  for 
each  of  the  basic  strategies  above  is 
programmed  into  a  separate  module.  The 
builder  of  an  expert  system  takes  the 
model  of  the  domain,  adds  the  reasoning 
modules  appropriate  for  the  application 
and  then  builds  the  operator/system 

interface  needed  with  the  outside  world 
to  run  the  system. 

Figure  3  illustrates  the  overall 

development  process. 

III.  Space  Support 
Appl icat ions 

Ford  Aerospace's  knowledge-based 
system  architecture  has  been  applied  to 
automate  several  functions  in  an  effort 
to  test  its  full  capabilities  in  being 
able  to  "reason  about"  and  "analyze"  new 
situations.  Illustrative  examples  and 
lessons  learned  are  presented  below: 

Satellite  Autonomy:  The  first  step 

in  achieving  satellite  autonomy  is  to 
automate  the  satellite  control  function 
on  the  ground.  In  many  ways  this  step  is 
more  difficult  than  the  final  step  in 
space,  because  the  ground  station  doesn't 
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always  have  continuous  contact  with  the 
satellite  and  hence  must  reason  with 
incomplete  Knowledge  about  what  happened 
between  passes.  Our  version  of  an  auto¬ 
mated  satellite  control  system  is  called 
5  TAR  PI.  AN .  Figure  4  illustrates  some  of 
the  user  interface  screens.  It  has  been 
developed  using  several  architectures: 
all  rule-based,  frame  based  and  PARAGON 
r e p r esen t a t ions  .  To  date  GPS  and  DSCS 
subsystems  nave  been  modeled  and  Known 
anomalous  situations  have  been  presented 
as  input  and  handled  by  the  system. 
Based  on  our  experience,  there  are  three 
areas  where  further  worK  is  necessary  at 
this  time  on  this  application. 

1.  The  satellite  design  engineers 

fine  that  time  critical  feedbacK  loops 
can  only  be  represented  accurately  if 
precise  definitions  of  the  concepts 
involved  and  their  relationships  are 
provided.  These  are  needed  to  taKe  full 
advantage  of  the  simulation  capability 
within  PARAGON.  We  are  considering 
provision  of  a  sec  of  preprogrammed  con¬ 

cepts  and  relationships  for  this  situa¬ 
tion  so  the  user  can  choose  the  one  that 
fits  his  design  characteristics  most 
closely.  This  approach  to  aiding  the 
design  engineer  is  non-trivial,  because 
each  feedbacK  loop  is  i nt er - re  la t ed  to 
other  unique  domain  Knowledge  being 
entered  by  the  engineer. 

2.  Our  generic  causal  diagnosis 

processing  module  sometimes  stops  at  the 
wrong  level  in  finding  the  cause  of  a 
problem.  I'll  use  an  example  to  explain 

what  I  mean.  When  you  get  in  your  car 
and  turn  the  Key,  you  expect  the  engine 
to  start.  If  it  doesn't,  you  note  that 

this  information  deviates  from  what  was 
expected  and  reason  about  possible  causes 
for  the  discrepancy.  A  liKely  cause  is  a 
dead  battery,  so  you  turn  on  the  light 

switch  and  note  that  the  lights  don't 

worK.  The  result  of  this  test  maKes  a 

dead  battery  even  more  liKely  as  a  cause, 
so  you  put  in  a  new  battery.  What  if  the 

engine  still  doesn't  start,  or  it  starts 

for  a  short  time  and  then  repeats  the 
anomalous  behavior?  The  cause  could  be  a 
short  in  the  system  which  causes  the 
battery  to  go  dead  or  the  engine  and 
lights  to  be  inoperable.  For  an  auto¬ 

mobile.  there  are  approaches  that  can  be 
raKen  to  test  for  malfunctions  that  would 
be  at  the  next  level  beyond  an  apparent 
cause.  Also,  since  batteries  don't  cost 
much  and  the  risK  is  low,  because  the 
system  probably  won't  be  harmed  if  a  new 
battery  is  installed,  the  KBS  should  maKe 
this  recommendation  and  only  looK  for 
another  level  of  malfunction  if  that 
recommendation  fails.  However,  satel¬ 
lites  operate  in  a  completely  different 
environment.  The  most  liKely  cause  of  a 
discrepancy  shouldn't  be  fixed  with  a 
replacement  until  the  risKs  and  next 


level  of  causes  have  been  evaluated. 
Therefore,  our  causal  diagnosis,  goal 
determination,  planning  and  corrective 
action  modules  must  all  worK  together 
iteratively  to  provide  the  most  appro¬ 
priate  correction  recommendation  for  an 
orbiting  satellite.  Different  domains 
will  require  different  levels  of  cause 
identification.  How  best  to  incorporate 
this  information  is  still  being  investi¬ 
gated. 

3.  Real-time  operation  is  still  to 
be  demonstrated.  Until  reasoning  issues 
and  domain  expert  interface  issues  are 
resolved,  it  doesn't  maKe  sense  to  con¬ 
centrate  on  real-time  considerations. 
Also,  parallel  hardware  systems  are 
becoming  more  powerful  every  day.  This 
medium  will  be  a  major  factor  in  achiev¬ 
ing  real-time  operations. 

Remote  Maintenance  of  Electronic 
Equipment :  The  systems  we  have  demon¬ 
strated  have  used  distributed  Knowledge 
bases,  since  the  information  available  at 
the  remote  location  is  restricted  to  the 
site,  while  the  information  at  the 
central  control  center  includes  global 
information  that  impacts  priorities  and 
constrains  corrective  action  a  1 ternat i ves . 

Digital  Design :  A  system  that 
performs  fault  isolation  on  a  digital 
board  has  been  prototyped.  Chips  ana  tne 
wires  connecting  them  were  modeled  as 
well  as  individual  behavior  (Note:  wires 
have  individual  behavior  patterns  such  as 
propagation  time,  delay  time,  resistance, 
etc.).  The  major  problem  encountered  was 
timing.  similar  to  the  feedbacK  loop 
timing  problem  discussed  under  satellite 
autonomy. 

NetworK  Control:  A  system  that 
models  a  communication  satellite  with  its 
individual  transponders  and  the  ground 
stations  that  maKe  up  its  networK  has 
been  prototyped.  If  a  communicat ion  linK 
isn't  working  as  planned,  the  system 
decides  whether  weather,  jamming,  loading 
or  equipment  failure  is  the  cause  and 
then  recommends  a  solution.  If  weather 
is  the  cause,  using  an  alternative  ground 
station  and  a  connecting  ground  link 
might  be  the  recommended  solution.  Time 
synchronization  is  a  consideration  in 
this  application.  PARAGON  currently 
models  timing  as  a  continuous  behavior. 
Further  enhancements  to  the  development 
interface  will  make  it  easier  tc  describe 
delay  times  and  cyclic  processes. 

Manufacturing  Plant  Process  Control: 
PARAGON  is  being  used  by  Ford  Motor 
Company  to  model  one  of  their  manufactur¬ 
ing  plants  and  simulate  the  process  (from 
the  top  down).  Used  only  as  a  simulation 
tool.  the  sensitivity  of  material  in 
process  and  inventory  to  schedule  can  be 
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4  evaluated.  Once  the  "reasoning"  modules 

are  added,  inefficient  situations  can  be 
»  detected,  the  cause  of  the  problem 

*  identified  and  recommendations  for 

improvement  made.  At  first  we  considered 
using  existing  domain  simulators  that 
represent  many  years  of  development 
effort,  but  the  knowledge  is  not  appro¬ 
priately  represented  for  a  reasoning 
module  to  use.  Translation  may  be  feas¬ 
ible  for  some  small,  simple  domains,  but 
not  for  large,  complex  domains.  The 
lesson  learned  is:  build  the  domain 

model  using  the  PARAGON  representation 
scheme,  test  and  V&V  the  knowledge  base 
using  the  model  as  a  simulator,  add 
generic  (or  domain  specific  if  necessary) 
reasoning  modules  that  have  been  tested 
and  V&Vd  separately,  build  an  interface 
with  the  external  world  (operator, 
telemetry  stream,  etc.)  and  you  have  a 
powerful  simulator  within  an  even  more 
powerful  KBS  that  will  meet  most  of  the 
desired  space  support  requirements  for 
KBSs  . 

PARAGON  currently  is  being  used  in  an 
R&D  and  staff  consultant  environment. 
Extensions  need  to  be  made  to  the  system 
to  increase  its  reasoning  capability. 
Real-time  applications  will  be  pursued  to 
test  its  parallel  reasoning  structure  on 
parallel  processing  hardware.  Maybe  next 
year  we  can  demonstrate  an  even  further 
narrowing  of  the  KBS  expectation  gap! 
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Abstract 

Work  on  providing  a  comprensive  validation  capability  tor  knowledge-based  systems 
has  been  progressing  successfully  at  the  Lockheed  Al  Center-Austm  for  more  than  a 
year  The  Expert  Systems  Validation  Associate  (EVA)  already  extends  and  comple¬ 
ments  the  validation  capability  provided  by  existing  commercial  or  academic  expert 
system  sheiis.  In  this  paper  we  outline  the  architecture,  theoretical  basis,  and  func¬ 
tionality  of  EVA  and  describe  its  existing  and  future  features.  The  existing  features  are 
currently  implemented  for  knowledge-based  systems  developed  in  ART  (Automated 
Reasoning  Tool,  Inference  Corporation). 


INTRODUCTION 

Knowledge-based  system  (KBS)  technology  has  now 
emerged  from  applied  artificial  intelligence  (Al)  as  a  vi¬ 
tal  technology  for  modeling  complex  systems  in 
defense,  industry,  business,  and  science. 

The  growing  reliance  on  KBSs  requires  the  develop¬ 
ment  of  appropriate  methods  for  validating  such  sys¬ 
tems.  The  obiective  of  the  Expert  System  Validation 
Associate  (EVA)  —  under  development  at  the  Lockheed 
Al  Center  at  Austin,  TX  -  is  to  define  and  develop 
automated  tools  to  validate  the  structural',  logical,  and 
semantic  integrity  of  KBSs.  In  this  paper,  which  is  a 
revised  and  expanded  version  of  a  previous  paper1,  we 
define  KBS  validation  as  the  verification,  validation,  and 
testing  of  a  KBS.  Our  definition  does  not  include  the 
evaluation  of  the  quality  of  a  system  by  a  group  of  ex¬ 
perts. 

Literature  on  validating  traditional  software  abounds 
(for  a  condensed  bibliography,  see  Miller2).  The  valida¬ 
tion  of  KBSs,  however,  has  received  considerably  less 
attention  in  the  Al  literature.  Exceptions  are  Nguyen  et 
al.3  and  Suwa  et  al.4. 

Validation  methods  for  traditional  software  are  difficult, 
time-consuming,  and  not  necessarily  correct.  We  have 
learned  to  use  the  expression  correct  with  respect  to 
specifications  rather  than  plain,  unrestricted  correct. 
Traditional  validation  -  represented  by  the  waterfall 
modeP  -  involves  the  complete  software  development 
life-cycle  and  is  not  oriented  towards  the  incremental 
development  method  typical  of  knowledge-based  sys¬ 
tems. 

During  the  construction  of  typical  KBSs,  requirements 


pertaining  to  “syntactic"  properties  of  rules,  clauses,  and  facts 
in  a  knowledge  base 
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are  most  of  the  time  not  as  precise  as  the  requirements 
for  a  conventional  software  development  project.  Thus 
the  system  designer  needs  validation  tools  io  assist 
him/her  m  clarifying  the  requirements  and  obtaining  a 
precise  specification.  This  is  typically  obtained  through 
rapid  prototyping  of  a  partial,  incomplete  specification 
Based  on  the  output  obtained,  the  specification  ,s 
modified  (corrected,  deleted,  or  extended),  until  the 
problem  appears  to  be  sufficiently  well  formulated  and 
understood.  An  appropriate  model  for  the  development 
of  such  systems,  the  spiral  model,  has  recently  been 
developed  by  Boehm6. 

A  second  reason  for  the  difficulty  of  applying  conven¬ 
tional  validation  methodology  to  a  KBS  is  that  a  KBS 
does  not  exhibit  the  functional  charactenstic  of  stan¬ 
dard  procedural  software  modules  where  given  input(s) 
are  mapped  into  specified  output(s).  A  KBS  is  mostly 
nonprocedural,  and  it  normally  does  not  contain  com¬ 
ponents  analogous  to  standard  software  modules 
amenable  to  validation  by  conventional  tools. 

It  is  our  conjecture  that  software  validation  can  be  more 
easily  performed  in  a  KBS  environment.  In  such  an  en¬ 
vironment  the  number  of  life  cycle  steps  is  reduced 
from  the  traditional  four  (requirements  development, 
specification  development,  design  development,  code 
development)  to  just  the  first  two,  resulting  in  a  con¬ 
siderable  reduction  in  the  amount  of  validation  work  to 
be  performed.  In  addition,  KBSs  can  be  regarded  as 
logic-based  systems,  consisting  of  a  declarative 
(application)  and  control  component  (inference  engine). 
The  logic-based  languages  used  to  implement  the  ap¬ 
plications  are  more  amenable  to  validation  using  logic 
and  mechanical  theorem  proving.  Finally,  the 
availability  of  metaknowledge  with  semantic  information 
and  integnty  constraints  permits  (a)  the  formulation  of 
conditions  (knowledge  base  des-nehs  specifications) 
that  the  rules  and  facts  of  a  KBS  nave  to  meet  and  (b) 


•Ke  automatic  checking  of  me  validity  cf  the  underlying 
aDpiicat'on. 

KBS  DEVELOPMENT  AND  VALIDATION 

T^e  rcremental  development  process  used  m  building 
KBSs  s  represented  m  hgure  ’ ,  adapted  from  Boehm's 
spirai  mccei. 

T"e  spiral  begins  with  a  set  of  mtial  requirements, 
describing  the  problem  to  be  solved  The  requirements 
onase  s  followed  by  the  specification  phase,  in  which 
:-e  expert  s  knowledge  about  the  domain  in  question  is 
translated  mto  rules  and  facts  Because  of  the  clear 
separation  between  declarative  and  control  component, 
these  specifications  can  be  executed  immediately, 
resulting  m  an  initial  operational  capability  (IOC),  a  first 
prototype.  The  expert  or  knowledge  engineer  repeats 
this  process  (goes  through  the  spiral),  expanding  or 
'“vising  the  requirements  and  specifications  until  after 
many  iterations  (with  many  interim  operational 
capabilities)  the  final  operational  capability  (FOC),  the 
end  product,  has  been  obtained. 

It  should  be  obvious  that  the  number  of  iterations  could 
oe  reduced  considerably  given  the  availability  of  an 
automated  validation  tool. 


EVA  GCAL 

The  goal  of  EVA  is  to  provide  au*cmated  too  s  m 
validate  the  logical,  structural,  and  semantic  "tegnty  c‘ 
KBSs.  In  addition.  EVA  also  addresses  the  gcai  of  re¬ 
proving  the  validation  process  by  fmcmg  mistakes  and 
emissions  in  the  knowledge  base,  proposing 
knowledge  base  extensions,  restrictions.  and 
generalizations  In  other  words.  EVA  addresses  net 
only  the  question  of  "Is  a  KBS  application  correct7"  put 
also  the  question  of  "Is  the  knowledge  usee  by  me  ex¬ 
pert  for  the  application  correct7" 

EVA  ARCHITECTURE 

To  permit  the  addition  of  future  functionality  EVA  ,s 
designed  to  be  a  metaknowledge-based  system  shell 
whose  architecture  is  shown  in  Figure  2 
Standard  commercial  expert  system  shells,  also  called 
object  shells,  provide  the  following  basic  constructs: 
objects,  classes  of  objects  (object  schemas), 
generalization  hierarchies  among  the  classes  of  ob¬ 
jects,  attnbutes  (slots)  of  objects,  property  inheritance, 
relation  schemas,  facts,  rules,  and  so  on.  Applications 
are  wnften  in  the  object  shells  by  application  program¬ 
mers  or  knowledge  engineers. 


Figure  1  Boehm's  spiral  model 


Figure  2.  EVA  architecture 


!n  contrast  to  the  knowledge  base  of  an  ob|ect  shell, 
EVA  provides  a  metashell  with  a  metaknowledge  base 
that  supports  higher  order  constructs  to  represent 
*.npwledge,  properties,  and  constraints  about  the  lower 
evei  obiects.  object  schemas,  relation  schemas,  the  or- 
denng  of  rules,  and  so  on,  of  an  application. 

EVA  FUNCTIONALITY 

The  purpose  of  our  research  effort  is  to  design  and  im¬ 
plement  a  very  expressive  metashell  based  upon 
-  gher-order  constructs.  For  example,  it  can  be  used  to 
'epresent  that  a  relation  is  symmetnc,  nonsymmetric, 
asymmetnc.  transitive,  nontransitive,  intransitive 
'ef,exive.  rrefiexive,  connected,  mandatory,  critical, 
synonymous  to  some  other  relation,  and  others.  The 
metasheil  can  be  used  to  represent  validation  state¬ 
ments  on  constraints  and  control  strategies  for  the  ap- 
p  cation  by  an  application  system  designer. 

The  ‘unchcnality  of  EVA,  represented  by  means  of  a 
da*a  flow  diagram,  .3  depicted  m  Figure  3. 


EVA  contains  two  kinds  of  functional  components 
modules  specific  to  a  particular  object  shell  (i.e  .  con¬ 
version  algorithms  used  by  the  analyzer)  and  modules 
specific  to  EVA,  the  validation  tools. 

The  analyzer  uses  conversion  algonthms  to  translate 
the  application  (represented  by  rules,  facts,  and 
schemas  of  the  object  shell)  and  the  validation  state¬ 
ments  into  the  EVA  format  to  be  used  by  the  validation 
modules. 

These  validation  modules  are  the  logic  checker,  struc¬ 
ture  checker,  semantics  checker,  omission  checker, 
rule  proposer,  behavior  venfier,  and  control  checker. 
The  logic  checker,  structure  checker,  and  behavior 
venfier  address  the  question  ’Is  the  application  wntten 
in  the  object  shell  correct?",  while  the  semantics 
checker,  omission  checker,  rule  proposer,  and  control 
checker  address  the  question  ’Is  the  knowledge  used 
by  the  expert  (knowledge  engineer)  for  the  application 
correct7"  For  example,  the  sentence  ’An  automobile 
does  not  need  motor  oil"  is  grammatically  correct,  but  it 
conveys  incorrect  information. 


Figure  3.  EVA  functionality 


We  have  implemented  considerable  portions  of  the 
EVA  functionality  for  ART-based'  expert  systems.  Our 
efforts,  so  far,  have  concentrated  on  the  implemen¬ 
tations  of  the  logic  checker,  the  structure  checker  and 
the  semantics  checker. 

LOGIC  CHECKER 

The  purpose  of  the  logic  checker  is  to  check  whether 

the  rule  base  of  an  application  is  consistent  and 
'•numencally  complete." 

Consistency 

The  logic  checker  checks  for  two  types  of  inconsis¬ 
tency  direct  and  indirect.  Two  rules  are  directly  incon¬ 
sistent  if  they  contain  mutually  exclusive  right-hand 
sides  (RHSs)  and  the  left-hand  sides  (LHSs)  are  equiv¬ 
alent  (identical,  or  one  LHS  subsumes  the  other).  In¬ 
direct  inconsistency  occurs  when  two  rules  whose 
LHSs  are  equivalent  derive  contradictory  RHSs  after 
rewnting  intermediate  RHSs. 

In  *he  following  examples,  rules  1  and  2,  and  rules  3,  4, 
and  5  are  inconsistent  and  are  reported  by  EVA. 


Automated  Reasoning  Tool,  inference  Corporation 


Direct  Inconsistency 

Rule  f :  STUDENT(x)aSTAFF(x) 

->UNIVERSITY_MEMBER(x) 

Rule  2:  STUDENT(x)->  ->UNIVERSITY_MEMBER(x) 
Indirect  Inconsistency 

Rule  3:  TEACHING_ASSISTANT(x)  -»TENURED(x) 
Rule  4:  TEACHING_ASSISTANT(x)  — >  STUDENT(x) 
Rule  5:  STUDENT(x)  ->  -.TENURED(x) 

Numeric  Completeness 

In  checking  for  numenc  completeness  the  logic  checker 
will  issue  a  warning  when  gaps  in  a  specified  numeric 
range  occur.  If  the  rule  base  contains  only  rules  6  and 
7  (dealing  with  the  number  of  items  in  a  discounted 
order), 

Rule  6:  DIS-ORDER(x)  a  <(NUM-ITEMS(x),50) 

-»  MIN-DISCOUNT(x) 

Rule  7:  DIS-ORDER(x)  a  >(NUM-ITEMS(x),75) 

-»  MAX-DISCOUNT(x) 

then  the  logic  checker  will  propose  the  rule  a  to  com- 
p'e'e  the  numenc  range. 

Rule  a:  DIS-ORDER(x)  a  ;  (NUM-ITEMS(x),50) 
a  <(Num-ltems(x),75)  ->  ?7 
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EXTENDED  LOGIC  CHECKER 

The  purpose  of  the  extended  logic  checker  is  to  checK 
‘or  inconsistencies  and  conflicts  caused  by  generaliza¬ 
tion  hierarchy,  incompatibility,  or  synonymy.  If  the  ap¬ 
plication  contains  rules  that  can  derive  contradictory 
conclusions  ‘rom  the  same  set  of  facts  and  from  the 
properties  of  generalization  hierarchy,  incompatibility, 
or  synonymy,  then  the  application  is  inconsistent. 

I nconsistency  Under  Generalization  Hierarchy 

Given  the  metafact  that  a  submanne  is  a  subclass  of 
ship,  rules  8  and  9  are  inconsistent. 

Rule  8  E  •  F  ->  ^send-ship. 

Rule  9  E.'F  send-submanne 

Inconsistency  Under  Incompatibility 

Given  the  metafact  that  boy  and  girl  are  incompat  ble, 
rules  1 0  and  1 1  are  inconsistent. 

Rule  10  AaB  -~is-boy 
Rule  1 1 :  AaBaC  — »is-girl. 

Inconsistency  Under  Synonymy 

Given  the  metafact  that  submarine  and  sub  are 
synonymous,  rules  12  and  13  are  inconsistent. 

Rule  12:  EaF  ->send-submanne. 

Rule  13  EaF  — >  -,send-sub. 

in  each  of  the  above  examples,  the  conditions  m  each 
of  the  rule  pairs  are  exactly  the  same  or  one  condition 
s  a  prcpe1-  suoset  of  another  However,  the  mconsis- 
'ency  can  occur  in  a  more  general  case  The  examples 
below  are  in  conflict  if  all  of  the  conditions  A,  B,  C, 
and  D  are  met  m  the  knowledge  base  at  the  same 
time 

Conflict  Under  Generalization  Hierarchy 

Gi  en  the  metafact  that  submanne  is  a  subclass  ot 
ship,  rules  14  and  1 5  are  in  conflict. 

r,He  14:  AaB  —»  ^send-ship. 

Rule  1 5  CaD  — ►  send-submanne. 

Conflict  Under  Incompatibility 

Gwen  the  metafact  that  boy  and  g<rl  are  incompatible 
ru!es  1 6  and  1 7  are  in  conflict 

Rule  16  A'8  — >.s-boy 
Rule  17  C^D-vis-girl 

Conflict  Under  Synonymy 

Given  the  metafact  ’hat  submarine  and  sub  are 
synonymous  rules,  ’8  and  19  are  in  conflict 


Rule  18  AaB  — * -send-suo 
Rule  19:  CaD  ->send-suorr.anne 

If  the  conditions  A,  B,  C,  and  D  can  be  satisfied  by  fu¬ 
ture  facts  in  the  application  (recognized  from  assertions 
,n  the  RHS  of  rules),  the  logic  cnecker  warns  of 
potential  conflict. 

STRUCTURE  CHECKER 

The  kernel  module  of  the  structure  checker  is  an 
analyzer  which  takes  in  the  facts  and  rules  of  a 
knowledge-based  application  and  builds  a  rule  graph 
An  arc  in  the  rule  graph  denotes  a  match  between  a 
relation  in  the  LHS  of  a  rule  and  a  relation  in  the  RHS 
of  a  ruie.  Other  structure  checker  modules  analyze  the 
rule  graph  to  locate  all  the  structural  errors  in  the 
knowledge  base,  such  as  nonreachability,  redundant 
rules.  irrelevant  rule  clauses,  and  rule  cycles. 

Reachability 

The  purpose  of  reachability  checking  is  to  determine  if 
the  rule  graph  has  any  missing  subgraphs  or  discon¬ 
nected  subgraphs. 

In  checking  for  reachability,  EVA  looks  for  dead-end 
nodes  and  unreachable  nodes.  A  node  here  means  an 
atomic  formula  in  a  rule.  A  ’dead  end"  occurs  when  a 
node  in  the  LHS  of  a  rule  does  not  match  any  fact, 
goal,  schema,  or  RHS  rule  node  in  the  knowledge 
base.  An  unreachable  node  is  a  fact.  goal,  schema,  or 
RHS  rule  node  that  cannot  be  matched  by  any  LHS 
rule  node  in  the  knowledge  base 

Redundancy 

A  rule  is  redundant  if  rt  is  duplicated  or  subsumed  by 
another  rule.  Two  rules  duplicate  one  another  if  they 
have  the  same  LHS  and  RHS  clauses,  possibly  in  dif¬ 
ferent  order  or  with  different  vanable  names.  Sub¬ 
sumption  occurs  when  two  rules  have  duplicate  RHSs 
and  the  LHS  of  one  is  a  subset  of  the  LHS  of  the  other, 
or  when  two  rules  have  duplicate  LHSs  and  the  RHS  of 
one  is  a  subset  of  the  RHS  of  the  other,  or  a  mixture  of 
both. 

Duplication: 

Rule  20:  MALE(x)a  PARENT(x)-»FATHER(x) 

Rule  21  PARENT(y)A  MALE(y)— >FATHER(y) 
Subsumption: 

Rule  22:  TENURED(x)a-,STAFF(x) 

-» UNIVERSSTY_MEMBER(X) 

Rule  23:  TENURED(x) ->UNIVERSITY_MEMBER(x) 


Relevance 

Two  rules  contain  relevant  or  superf'ucus  causes  f 
they  are  duplicates  except  that  .n  one  LHS  a  clause  p 
‘ s  affirmed,  in  the  other  denied. 

The  structure  checker  reports  that  the  f: rst  clause  m 
rules  24  and  25  is  irrelevant. 

Rule  24.  TEACHING  ASSISTANTDDSTAFFfx) 

— >  UNIVERSITYMEMBERD) 

Rule  25:  ^TEACHING_ASSISTANT(x)aSTAFF(x) 

—  UNIVERSiTY_MEMRER(x) 

Cyc'es 

EVA  checks  for  direct  and  indirect  cycles  in  the  rule 
base.  A  direct  cycle  exists  when  the  same  clause  oc¬ 
curs  on  both  the  LHS  and  RHS  of  a  rule.  This  may 
cause  the  rule  to  fire  repeatedly  --  that  is.  result  in  an 
nfimte  loop  In  the  case  of  an  indirect  cycle  the  infinite 
oop  invokes  two  or  more  rules. 

Direct  Cycle: 

Rule  26.  ANCESTOR(y,x)APARENT(x.z) 

—  ANCESTOR(y.z) 

Indirect  Cycle. 

Rule  27:  HU MAN D)  —>  PERSON(x) 

Rule  26:  PERSON(x)  — > HUMAN(x) 

EXTENDED  STRUCTURE  CHECKER 

The  extended  structure  checker  checks  for  duplication, 
subsumption,  relevance,  and  cycles  caused  by 
generalization  hierarchy  or  synonymy. 

Duplication 

Given  the  metafact  that  sub  and  submanne  are 
synonymous,  rules  29  and  30  are  duplicates  of  one 
another 

Rule  29:  SUBDi  -  DIVED) 

Rule  30.  SUBMARINED;  —  DIVE(x) 

Subsumption 

Given  the  metafact  that  submanne  s  a  subclass  of 
ship,  njie  3i  is  subsumed  by  rule  32,  i.e  ,  whenever  31 
s  applied.  32  will  also  be  applied. 

Puie  31  SUBMARINED)— » LAUNCH  (x) 

Rule  32  SHIPD)  —>  LAUNCH  D) 

Relevance 

Given  the  metafact  that  sub  and  submanne  are 
synonymous,  the  first  clause  m  rules  33  and  34  is  ir- 
re!evant 


Ru:e  33  SUBMARlNE'xj-Fxi  — E  < 

Rule  34  - SUBi x;  -  F  xi Ei x) 

Indirect  Cycle 

Giver,  the  metafact  that  sub  and  suomanne  are 
synonymous,  rules  35  and  36  create  an  indirect  cyc  e 

Rule  35.  SUBMARINED) —*  GD) 

Rule  n6:  G(x)  — >SUB(x) 

SEMANTICS  CHECKER 

The  semantics  checker  has  two  ma]or  functions  check¬ 
ing  the  apDiication  written  m  the  object  shell  for  viola¬ 
tions  of  the  semantic  constraints,  and  checking  tne 
semantic  constraints  themselves  for  internal  consis¬ 
tency  and  agreement  with  other  metaconstramts 
represented  in  the  metashell. 

EVA'r  metarelations  take  object,  class,  sic*  and  "ela¬ 
tion  names  in  the  object  sheil  as  argumen.s.  they  per¬ 
mit  tne  knowledge  base  designer  to  specify  seman-'c 
constraints  (conditions)  that  must  be  met  by  the  under- 
'ymg  application.  Facts  violating  the  semantic  fact  con¬ 
straints  are  either  incorrect  facts  or  snow  that  the 
metaconstraint  itself  is  net  correct,  wm.e  rules  violating 
the  semantic  rule  constraints  are  either  incorrect  rules 
or  show  that  the  metaconstraint  is  not  correct 

In  particular,  the  semantics  checker  checks  facts  in  the 
application  for  violation  of  range  constraints,  mini¬ 
mum/maximum  cardinality  constraints,  legal  value  con¬ 
straints.  value  compatibility  constraints,  and  for  other 
semantic  constraints  such  as  subreptions,  inverses, 
and  data  types. 

The  semantic  constraints  that  have  been  implemented 
m  an  ART-based  prototype  are  explained  Ke!ow.  More 
semantic  constraints  -  circular  ranges  (Sunday, 
Monday,...;  00:00-24:00  hours;  Jan...  Dec;  and  so  on), 
and  nonnumenc  ranges  (cold,  warm,  hot;  Pnvate,  Cor¬ 
poral,  Sergeant,...;  and  so  on),  as  well  as  relations  be¬ 
tween  relations/attnbutes/sets  such  as  inverse, 
synonymous,  antonymous  (male/female,  i.e.,  not  male 
implies  female  and  vice  versa),  contrary  (young/old,  i.e. 
not  young  does  not  imply  old),  compatible,  incom¬ 
patible,  and  others  -  will  be  implemented  in  1987. 

lower_upper(slot,  class,  lower,  upper) 

This  metarelation  defines  the  legal  range  of  numerical 
values:  the  values  for  the  <slot>  of  the  <class>  must 
be  between  <lower>  and  <upper>.  EVA  discovers  and 
flags  values  that  exceed  these  bounds 


EVA  "oi  c":y  checks  ‘c  .ts  against  semantic  con¬ 
straints.  Out  a'SO  checks  that  tne  semantic  constraints 
•nemseives  are  consistent  Smce  a  CHILD  is  a  PER¬ 
SON.  tne  age  rarge  cf  CHILD  must  'ail  within  the  age 
range  of  PERSON  ThuS  me  ‘o  lowing  semantic  con- 
st'amts  would  oe  '•cons. stem 
is_a  CH'LD  PERSON , 
lower_upper  AGE  PFR30N.  i  IQ) 
lowerupper AGE.  CHILD  0.  ’2) 

legal_value(slot.  class,  values) 

This  metareiation  defines  the  legal  values  of  a  slot:  the 
values  for  the  <slot>  of  the  <dass>  must  be  listed  in 
<va!ues> 

For  examoie.  the  'oilowing  semantic  con  stramt 
legal_value;GENDER  STUDENT, 

[male,  female,  hermaphrodite]) 
states  that  students'  genders  must  be  male,  female,  or 
hermaphrodite.  EVA  flags  any  STUDENT  record 
where  GENDER  has  a  nonlegai  value. 

relation(rel,domain-1,...,domain-n) 

This  metareiation  defines  both  the  number  of  legal  ar¬ 
guments  of  a  relation  <rel>  and  the  legal  data  type  of 
each  argument:  <domam-i>  is  either  a  class  of  objects 
or  a  set  of  values  for  i=l . n. 

This  Kind  of  semantic  constraint  is  used  to  permit  EVA 
to  enforce  strong  data-type  checking  for  relations. 

For  example,  given  the  two  facts 
PERSON(charlie) 
and 

DOG(snoopy) 
and  the  metafact 

relation(MURDERER_OF, PERSON, PERSON) 

EVA  flags  as  erroneous  the  fact 
MURDERER  _OF(charlie, snoopy). 

compatible  arg_types(rel,argument_value_list) 

This  metareiation  defines  the  legal  value  combinations 
'or  individual  argument  types.  Thus  the  metafact 
compatible_arg_types(ENROLL, 

{(argl  (FRESHMAN) 

arg2  (Math 1 01  .English  101 )] 

[argl  (SOPHOMORE) 

arg2  (Math20l  ,Art201 ,...)] 

/) 


v.ates  teat  a  FRESHMAN  can  cn.y  ENROLL  n 
Math  101  English  101.  that  a  SOPHOMORE  can  only 
ENROLL  ,n  Math20l .  Art20l .  .  and  so  on 

EVA  also  checks  the  defined  inverse  R'  of  a  nation  R 
for  agreement  with  argument  data  type  specifications 
and  legally  of  argument  value  combinations  using  the 
metarules  stated  for  R 

min_max_rel(relation,  domain,  min,  max) 

This  metareiation  specifies  the  minimum  and  maximum 
number  of  tuples  (records)  of  a  relation  the  number  of 
records  of  <relation>  with  object  in  <domam>  must  be 
between  <mm>  and  <max>. 

For  example,  the  following  semantic  constraint 
min_max_rel(ENROLL,  STUDENT,  0,  5000) 

means  that  up  to  5000  student  enrollments  are  allowed 
m  the  data  base  at  any  one  time  The  enrollments  are 
represented  by  records  or  tuples  of  the  relation  EN¬ 
ROLL. 

min_max_role(relation,  domain,  min,  max) 

This  metareiation  defines  that  each  object  in  <domam> 
must  have  at  least  <min>  and  at  most  <max>  objects 
for  the  relation  <relation>. 

Thus  the  semantic  constraint 

min_maxjrole(ENROLL,  SOPHOMORE.  3,  4) 

states  that  each  SOPHOMORE  must  ENROLL  in  at 
least  3  and  at  most  4  courses.  EVA  discovers  any 
sophomore  who  enrolls  in  fewer  than  three  or  more 
than  four  courses. 

incompatlble(s!ot,  class,  values) 

This  metareiation  defines  illegal  value  combinations: 
th6  multivaiue  <s,ot>  or  <class>  must  not  have  the 
combinations  listed  in  <values>. 

For  example,  the  semantic  constraint 

incompatiblefTITLE,  UNIVERSITYMEMBER 
[(Tenured, Untenured), 

(Tenured, Student), 

(Tenured, Staff)]) 

means  that  no  UNIVERSITY  MEMBER  can  have  both 
the  TITLE  Tenured  and  Untenured,  Tenured  and  Stu¬ 
dent,  or  Tenured  and  Staff. 

compatlble(slot,  class,  values) 

This  metareiation  defines  legal  value  combinations: 
the  multivalue  <slot>  of  <class>  may  contain  the  com¬ 
binations  listed  in  <values> 


For  example,  the  semantic  constraint 
compatible(TITLE.  UNIVERSITY_MEMBER 
[Untenured.  Student.  Staff]) 

means  that  a  UNIVERSITY_MEMBER  may  have  up  to 
three  TITLES  from  the  list  Untenured,  Student,  or  Staff. 
[This  metareiation  complements  the  previous  one  and 
is  to  oe  used  when  the  number  of  illegal  combinations 
woi  "ct  ne  too  tedious  to  list.] 

subrelation(relation1,  relation2) 

This  metareiation  defines  that  <relation1>  is  a  subrela- 
t ion  of  <relation2>.  EVA  checks  that  the  number  of  ar¬ 
guments  for  <reiationt>  and  <relation2>  are  the  same, 
ana  that  the  data  types  of  the  arguments  of  <relationl> 
are  subclasses  of  the  corresponding  arguments  of 
<relation2> 

For  example.  EVA  determines  that  the  semantic  con¬ 
straints 

relation(KILLER_OF, 

ANIMATEjDBJ,  ANIMATEjDBJ) 
relation!  MU  RDE  RE  R_OF, 

PERSON,  THING) 

subrelation(MURDERERjDF.  KILLER_OF) 
are  inconsistent  since  the  second  argument  (THING)  of 
MURDERERJDF  is  not  a  subset  of  the  second  ar¬ 
gument  (ANIMATEjDBJ)  of  KILLERJDF 
EVA  also  checks  that  the  inverse  of  <relation1>  is  a 
subrelation  of  the  inverse  of  <relation2>.  EVA  also 
creates  the  missing  inverse  of  a  subrelation,  if  one 
does  not  exist 

FUTURE  EVA  FUNCTIONALITY 

The  functional  components  described  below  are  part  of 
our  current  research  and  will  oe  implemented  in  future 
EVA  prototypes. 

Omission  Checker 

Very  often  an  application  wntten  in  the  object  shell  is 
incomplete.  This  means  there  are  omissions  in  the  ap¬ 
plication  knowledge  base.  The  omissions  may  be  re¬ 
nted  to  the  structural  rule  graph,  e  g.,  omission  of  rules 
for  missing  cases  or  omission  of  terminating  rules. 
These  are  similar  to  missing  some  case  or  terminating 
statements  in  a  conventional  programming  language. 

The  other  facet  of  the  omission  checker  is  related  to  in¬ 
duction  or  machine  learning.  Through  the  recognition  of 
structural  information  contained  in  facts  and  rules  in  the 
application  and  semantic  information  contained  in  the 
metaknowledge  base,  the  omission  checker  may  iden¬ 
tify  some  relations  f"  at  are  not  defined  by  the  user,  or 


earn  rules  to  associate  certain  conditions  wth  " 
classes  of  objects. 

For  example,  if  the  following  information  is  known 
Class  of  objects:  person,  man.  woman. 

Generalization  hierarchy: 

man  isa  person,  woman  isa  person, 

Metafact: 

person  is  union  of  man  and  woman; 

Relation  schema: 

PARE  NT-OF(person,  person), ■ 

Relation  schema 

FATHER-OF(man.person), 

Metafact: 

FATHER-OF  is  subrelation  of  PARENT-OF, 

then  the  emission  checker  can  prompt  the  user 
whether  there  should  be  a  relation  that  relates  woman 
.nd  person,  in  analogy  to  FATHER_OF  whicn  re  ates 
man  and  person,  (namely,  MOTHER-OF). 

Rule  Proposer 

There  are  two  ways  to  use  the  rule  proposer.  The  user 
can  use  it  to  simplify  a  collection  of  rules  througn  induc¬ 
tion.  That  is.  some  conditions  appeanng  in  a  collection 
of  rules  may  be  abstracted  by  induction  into  a  more 
general  condition,  and  thus  the  collection  of  rules  can 
be  simplified  into  a  single  rule.  For  example,  in  njles 
37,  38,  and  39, 

Rule  37:  LOSS_CANOPY(x)aALTITUDE(x) >20000 
-♦REDUCE-ALTITUDE(x). 

Rule  38:  LOSS_OXYGEN(x)aALTITUDE(x)> 20000 
^REDUCE-ALTITUDE(x). 

Rule  33:  LOSS_ENGINE(x)aALTITUDE(x)  > 20000 
->  REDUCE-ALTITUDE(x). 

the  relations  dealing  with  the  "loss'  of  a  system  part  are 
abstracted  into  a  more  general  relation,  e.g.,  CRIPPLE. 
Rule  a:  LOSS_CANOPY(x)-»CRIPPLE(x) 

Rule  0:  LOSS_OXYGEN(x)-*CRIPPL£(x) 

Ruler  LOSS_ENGINE(x)-»CRIPPLE(x) 

and  the  three  rules  (37,  38,  39)  are  reduced  to  one 
general  rule. 

Rule  p:  CRIPPLE(x)aALTITUDE(x)> 20000 
-»  REDUCE-ALTITUDE(x). 

On  the  other  hand,  the  rule  proposer  can  be  used  in  an 
intelligent  debugger  to  adjust  rules  to  fit  test  cases 
When  the  application  programmers  run  the  application 
on  the  test  cases,  they  may  find  errors  in  the  applica¬ 
tion  caused  by  incorrect  rules.  Conditions  in  the  LHS 
of  a  rule  usually  define  a  class.  In  the  case  of  an  incor¬ 
rect  rule,  the  rule  proposer  may  apply  the  restnction  or 
generalization  operation  on  the  rule  to  define  a  more 


specific  or  more  general  class,  respectively.  The 
modified  rule  may  allow  the  application  to  pass  the  test 
cases  successfully 

Behavior  Verifier 

A  system  may  De  decomposed  into  many  subsystems 
A  subsystem  may  be  represented  by  a  collection  of 
‘jets  and  rules  m  the  object  shetl  However,  the  suo- 
system  must  have  external  mput/output  interfaces  to 
communicate  with  the  outside  world.  For  example,  in 
the  space  shuttle  flight  software  system,  the  navigation 
controller  is  a  subsystem  that  sits  in  a  control  loop,  col¬ 
lects  and  analyzes  data,  and  then  sends  control  signals 
to  the  vehicle  manipulator. 

The  behavior  of  the  subsystem  is  a  description  of 
relationships  among  the  external  mput/output  interfaces 
and  internal  states  of  the  subsystem.  The  subsystems 
are  connected  together  to  make  the  system  The 
verifier  s  to  prove  that  the  intended  behavior  of  the 
system  can  be  derived  from  the  behaviors  of  the  sub¬ 
systems  and  the  connection  descriptions. 

Control  Checker 

Ail  existing  object  shells  provide  some  control  con¬ 
structs  to  sequence  the  firing  of  rules  in  the  application. 
For  example,  in  ART,  facts  acting  as  permitting  con¬ 
ditions  and  salience  (pnonty  instructions)  can  be  used 
to  control  the  rule  firing  sequence. 

In  EVA,  if  the  system  designers  want  to  impose  order- 
ng  constraints  on  some  actions,  they  can  specify  them 
n  the  metashetl,  and  the  control  checker  will  verify  if 
the  order  specified  m  the  application  rules  corresponds 
to  or  violaies  the  ordenng  constraints  specified  in 
EVA's  metacomponents  For  example,  in  an  office  sys¬ 
tem,  an  ordenng  constraint  is  that  a  paper  must  be 
"cleared"  before  it  is  "published." 

CONCLUSION 

The  first  prototype  of  EVA  has  been  implemented  in 
ART  and  LISP  ART-based  systems  being  developed 
at  Lockheed  such  as  the  Software  Project  Management 
system7  nave  been  used  as  test  cases.  Other 
Lockheed  companies  are  using  our  prototype  for  their 
Knowledge-based  applications. 

It  is  evident  that  EVA  provides  a  powerful  means  for 
representing  knowledge  about  an  application  domain 
and  for  venfymg  that  the  knowledge  is  correct  and  com¬ 
plete  EVA  increases  the  reliability  of  knowledge- 
based  systems,  speeds  up  their  development,  and  as¬ 
sists  in  their  continuing  modification.  The  necessity  for 


such  validation  tools  will  continue  to  grow  as  'uture 
knowledge-based  systems  play  a  more  critical  rcie  in 
business,  industry,  government,  and  the  sciences 
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ABSTRACT 

We  have  taken  up  the  challenge  of  integrating 
telemanipulation  technology  and  autonomous  system 
technology.  We  are  seeking  methods  for  integration 
at  a  fundamental  rather  than  an  ad-hoc  level.  We 
believe  that  success  in  this  effort  can  open  up  new 
space  and  defense  applications  now  beyond  reach  of 
either  technology  alone. 

In  our  presentation  at  this  symposium,  we 
introduce  a  senes  of  concepts  for  "tele-autonomous'' 
systems.  The  concepts  involve  new  system 
architectures  and  associated  new  system  interface 
controls  including  "time  clutches",  "position 
clutches"  and  "time  brakes". 

Taken  together,  the  concepts  enable  effective, 
efficient  intermingling  of  real-time  cognition  and 
manipulation  tasks  performed  by  either  humans  or 
machines.  The  concepts  also  yield  simple  mechanisms 
and  protocols  for  handoffs  of  such  tasks  between 
multiple  agents. 

In  this  presentation  we  focus  primarily  on  the 
tutorial  introduction  of  the  basic  tele-automation 
concepts.  We  then  briefly  describe  our  environment 
for  exploring  this  new  technology  and  the  results  of 
our  initial  experiments  Further  details  concerning 
tele-autonomous  syste  .  architecture  and  our  initial 
experimental  results  can  be  found  in  an  attached 
reference  [CON87). 


•  This  presentauon  is  based  on  recent  work  described  in  a 
paper  [CON 87]  to  be  published  in  the  Proceedings  of  the  IEEE 
Intemauonal  Conference  on  Roboocs  and  Automation,  March 
30.  1987.  A  preprint  of  that  paper  is  included  with  this 
A1A A/NAS A/L'SAF  Symposium  preprint  Our  presentation  is 
further  supplemented  with  a  University  of  Michigan  Robotics 
Research  Laboratory  video-report  (CON87a|. 
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INTRODUCTION 

We  are  seeking  simple,  generic  methods  for 
intermingling  and  in'egrating  telemanipulation  and 
autonomous  systems  technology.  Now,  you  might 
ask,  why  would  we  want  to  do  that? 

First,  we  wish  to  provide  more  effective  systems 
for  autonomous  environmental  manipulation. 
Consider  an  AI  cognition  system  embedded  within  an 
overall  perception-cognition-action  system.  Many 
tasks  of  interest  will  involve  perception-  cognition- 
action  computing  delays  on  the  order  of  fractions  of  a 
second  or  seconds.  How  can  we  deal  with  such 
delays,  when  the  basic  behavioral  acts  to  be  done  to 
complete  a  manipulation  task  themselves  require  only 
on  the  order  of  fractions  of  a  second  or  seconds? 

Presently  we  require  substantial  environmental 
knowledge  and  then  piece  together  preprogrammed 
forms  of  interactions  to  cope  with  such  delays.  When 
that  isn’t  possible,  we  fall  back  on  a  rather  halting, 
stumbling  form  of  perceive-think-act  cycling,  where 
the  perception  to  action  delays  arc  contained  in  each 
basic  behavioral  act.  Could  we  get  around  this 
somehow?  After  all,  animals  often  perform 
manipulations  with  the  aid  of  visualizations  out  just  in 
front  of  their  real  time  actions.  Could  we  mechanize 
something  like  that? 

The  second  challenge  is  to  provide  protocols  for 
the  interaction  between  multiple  autonomous 
manipulation  agents.  Consider  an  ALV  driving  down 
a  remote  road.  It  suddenly  encounters  uncertain 
footing,  and  doesn't  have  sufficient  exploratory 
behaviors  and  learning  capabilities  to  get  itself  out  of 
trouble.  We  know  that  AI  will  not  soon  be  able  to 
handle  all  the  cognitive  tasks  and  especially  not  all  the 
manipulation  tasks  to  get  an  ALV  out  of  this  kind  of 
trouble.  But  how  can  we  enable  a  human  to  easily 
"slip  into  the  cockpit”  and  take  over  in  mid-manuever 


in  such  situations1  How  are  they  to  perceive  the  local 
task  goals  and  judge  whether  they  are  making 
progress  towards  them1  And  how  are  they  to  hand 
the  task  back  to  the  machine1 

This  intriguing  example  illustrates  the  need  to 
bridge  the  gap  between  direct  human  control  and  AI 
control  of  manipulation  systems,  and  the  need  for 
handoff  protocols  between  autonomous  agents, 
whether  human  or  machine.  It  is  suggestive  of  the 
larger  space  of  such  examples,  many  of  which  are 
emerging  in  the  machine  intelligence  research 
programs  now'  underway  under  DARPA's  Strategic 
Computing  Initiative  [DAR83],  and  emerging  as  a 
result  of  NASA  space  station  automation  activities. 

The  basic  challenge  in  such  examples  is  how  to 
structure  the  actions  and  interactions  of  intelligent, 
goal-seeking  sy  stems  when  their  activities  are  at  least 
in  pan  based  on  their  physical  manipulations  of  their 
environment  rather  than  simply  on  symbolic 
communication.  The  challenge  is  how  to  mediate  the 
interactions  of  cognitive  agents  that  are  embedded  in 
perception-cognition-action  systems. 

Certainly  there  are  a  lot  of  ad-hoc.  task-specific 
systems  being  built  that  do  interact  effectively  in  very 
narrow  contexts.  But  such  ad-hoc  systems,  while 
individually  useful,  do  not  readily  generalize  to 
provide  a  basis  for  others  to  build  upon.  Could  we 
find  some  simple,  general  ways  to  think  about  the 
problem?  Could  there  be  some  basic,  generic 
protocols  on  top  of  which  we  could  construct  more 
uniform  systems?  We  think  the  answer  to  these 
questions  is  yes. 


BASIC  TELE-AUTOMATION  CONTROLS 

We  now  present  a  sequence  of  interface  control 
concepts  that  collectively  enable  efficient  control  of 
manipulation  tasks  and  that  enable  simple  protocols 
for  exchange  of  such  tasks  between  control  agents. 
This  paper  focusses  on  tutorial  development  of  the 
basic  functional  ideas.  For  details  on  architectures  to 
mechanize  these  concepts,  see  [CON87], 

We  begin  by  looking  via  video  link  over  the 
shoulder  of  a  telerobotic  manipulator,  and 
controlling  the  manipulator  via  a  joystick  as  shown  in 
Figure  1 .  We  are  to  perform  the  simple  task  of 
touching  in  sequence  each  of  a  series  of  boxes.  This 
task's  difficulty  is  some  function  of  the  ratio  of  the 
distance  between  consecutive  boxes  and  the  sizes  of 
the  bo~es.  The  difficulty  can  be  varied  easily,  and  we 
can  undertake  various  trials  of  performance  as  a 
function  of  system  parameters.  For  example,  we 
could  do  some  simple  trials  to  see  if  the  time  to 
complete  the  task  is  a  logarithmic  function  of  D/S,  as 
in  Fitt  s  Law  ICAR83]. 


Figure  1.  Visualizing  a  remote  manipulation  task 


Coping  with  Time  Delay 

Because  of  our  interest  in  remote  spate  and 
defense  systems,  we  visualize  trying  the  same 
manipulation  with  a  time  delay  inserted  into  the 
return  video  path.  We  find  that  the  telerobot's 
motions  then  tend  to  be  rather  slow  and  jerky.  The 
operator  must  move  a  little  and  then  wait  through  the 
time  delay  to  see  what  happened.  As  an  illustration  of 
this  point,  observe  in  the  video-report  the  telerobot 
performing  the  manipulation  task  with  no  delay  and 
then  with  a  2.0  second  delay.  The  difficulties 
introduced  by  the  time  delay  are  quite  noticeable. 
The  time  to  complete  the  task  is  greatly  extended. 
(Note  that  for  convenience,  we  display  a  model  of  the 
telerobot  using  a  Silicon  Graphics  IRIS  workstation. 
The  model  is  driven  by  the  actual  joint  angles  of  the 
telerobot,  and  is  thus  equivalent  to  observing  video  of 
the  real  telerobot  for  our  purposes  here). 

To  overcome  the  time  delay  problem,  Noyes  and 
Sheridan  [NOY84]  have  suggested  that  the  operator 
control  a  local  simulation  of  the  telerobot,  with  the 
control  signals  then  sent  in  parallel  to  the  simulation 
and  the  remote  telerobot.  The  simulation  is  then 
displayed  superimposed  over  the  return  video.  In  this 
way  the  operator  can  "see"  the  effects  of  the  control 
immediately  without  having  to  fully  wait  for  the 
return  signal  from  the  telerobot.  As  a  result,  task 
time  is  reducid  to  nearly  that  of  the  no-delay  case. 
(Noyes's  and  Sheridan's  concept  is  further  sketched 
in  Figures  2.1.  and  2.2  in  [CON87]). 
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Figure  2  presents  a  visualization  of  telerobotic 
manipulation  using  a  forward  simulation  to  cope  with 
the  time  delay,  as  does  the  next  segment  in  the 
video-report.  The  wire  frame  is  the  forward 
simulation  that  directly  responds  to  operator  control, 
and  the  solid  frame  represents  the  time  delayed  image 
of  the  real  telerobot.  Much  faster  and  smoother 
control  is  achieved,  as  is  evident  in  the  videotape. 
This  is  a  first  step  towards  evolving  machine 
manipulation  visualization,  since  the  visualization 
could  help  cope  not  only  with  communication  delays, 
but  also  with  computational  delays  within  a 
self-contained  autonomous  agent. 


□  3 
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Figure  2.  Visualizing  manipulation  through 
a  time  delay  using  forward  simulation. 


The  Time  Clutch 

But,  this  First  exploitation  of  real-time  forward 
simulation  is  only  the  beginning.  Forward  simulation 
can  also  be  exploited  even  if  we  don't  have  a 
communications,  time  delay.  To  do  this,  we  introduce 
the  concept  of  a  "time  clutclV'  to  disengage  synchrony 
between  operator  specification  time  and  telerobot 
manipulation  time  during  path  specification.  Our 
hypothesis  is  that  operators  can  often  think  of  and 
generate  a  path  segment  more  quickly  than  the 
telerobot  can  follow  it.  Once  generated,  such  a  path 


segment  can  then  be  followed  more  quickly  by  the 
robot  than  would  be  the  case  if  the  robot  were 
time-synchronized  to  the  specification  process:  with 
time  synchrony  disengaged,  the  robot  can  steadily 
proceed  dPnearly  its  maximum  rate,  subject  of  course 
to  error  Ifrtfits  and  hard  constraints. 

An  overall  diagram  of  the  basic  system 
architecture  including  the  time  clutch  is  contained  in 
figure  3.1  in  [CON87],  Figure  3  in  this  presentation 
shows  a  path  being  generated  well  out  in  advance  of 
the  actual  robot  by  an  operator  using  forward 
simulation  with  time  clutch  disengaged.  The 
associated  video-report  also  demonstrates  the  effects 
of  disengaging  the  time  clutch;  if  you  put  a  stop  watch 
on  the  action,  you  will  measure  a  significant  speed  up 
of  the  real  telerobot's  motion  from  that  obtained 
using  forward  simulation  alone. 

This  step  in  the  evolution  of  machine 
manipulation  visualization  enables  the  cognitive  agent 
to  "look  and  think  ahead"  of  the  manipulation  under 
control,  with  the  look-ahead  time  being  elastic,  and 
not  just  a  fixed  internal  or  external  system  time  delay. 
The  implementation  of  this  new  capability  requires 
only  a  simple  mutation  of  the  forward  simulation 
previously  used  for  coping  with  a  time  delay. 


Figure  3.  Rapid  manipulation  path  generation  using 
forward  simulation  with  time  clutch. 
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dutch 


The  P*sU*a 

We  next  introduce  the  concept  of  a  "position 
dutch"  which  enables  a  disengagement  of  position 
synchrony  between  simulator  and  manipulator  path 
(see  figure  3.1  in  [CON87]  for  system  diagram).  We 
hypothesize  that  faster,  shorter,  cleaner  paths  can  be 
generated  on  difficult  tasks  using  this  control.  This 
idea  is  illustrated  in  Figure  4,  which  shows  the  use  of 
the  position  clutch  to  disengage  from  path  generation 
during  a  dose  approach  to  a  difficult  manipulation 
(in  thtt  case,  touching  a  small  object). 


Figure  4.  Using  the  position  dutch  to  cope  with 
a  more  difficult  manipulation. 

Suppose,  for  example,  that  the  operator  had 
arrived  (in  the  simulation)  at  point  A  ahead  of  time 
by  using  the  time  clutch.  The  position  dutch  can  then 
be  disengaged,  stopping  the  output  from  the  operator 
control  from  going  to  the  real  telerobot  -  -  it  will 
only  go  to  the  simulation.  When  the  forward 
simulator  is  in  good  position,  the  position  dutch  will 
be  reengaged,  causing  a  short,  smooth  path  to  be 
inserted  that  links  to  the  earlier  path.  This  avoids 
inclusion  of  jittery  preposi honing  movements  in  the 
final  path  to  be  followed.  Further,  the  time  spent  by 
the  operator  m  achieving  the  proper  position  will  not 
be  incurred  by  the  real  telerobot  since  these  motions 
were  "clipped"  out  of  the  path  seta  to  the  telerobot. 


The  openaar  has  thus  used  up  same  of  tie  nme 
saved  through  use  erf  the  tone  with  die  result 

that  the  overall  task  time  of  the  telerobot  is  reduce 
still  further.  The  next  segment  in  the  video-report 
illustrates  this  pan*.  This  level  of  manipulation 
visualization  corresponds  to  rp*irfr  "vtsnahzancms  and 
visualized  trials  of  rrm  triple  alternatives"  prior  to 
committment  to  action,  and  its  implementation 
requires  only  another  simple  mi  nation  of  tk:  basic 
forward  sumikrion  capability. 

The  Tone  Brake 

To  handle  rfnimgranw  and  emus  we  introduce 
the  concept  erf  a  time  brake.  Thu  control  can  be  used 
to  deal  with  situations  such  as  something  failing  over 
a  previously  generated  pak,  as  ilk  grated  by  tlx  "X" 
m  Figure  5.  k  Figure  5  we  see  the  tone  brake  being 
applied  and  the  forward-emulated  manipulator 
backing  down  the  path  (a  a  race  to  get  cm  the  other 
side  of  the  obstacle  before  the  real  system  gets  there). 
See  figure  3.1  and  the  text  in  [CON87]  for  tin 
associated  system  arrhitrmnaJ  concepts.  The  next 
segment  of  the  video-report  demonstrates  the 
application  of  the  rime  brake. 


Figure  5.  Using  time  brake  to  handle  a  contingency. 
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This  aspect  of  TrsaafnaPm  corresponds  to  seeing 
sonoetfessg  a  bool  to  happen  that  will  interrupt  an 
action  previously  visualized  but  not  yet  underway.  If 
it  had  gotten  underway,  or  is  allowed  to  get 
underway,  the  system  will  have  to  deal  with  it 
through  local  reflex  action  or  crash.  But  if  visualized 
in  tune,  the  cognitive  agent  can  withdraw  the  action 
using  the  rime  brake. 

TASK  HAN  DOFFS  AND  RENDEZVOUS 

These  basic  tele-autonomous  system  interface 
controls  enable  ns  to  greatly  improve 
telemanipulation  performances,  as  we  ll  see  in  the 
discussion  of  our  initial  experimental  results.  But  the 
controls  do  more  rhan  that.  They  also  provide  the 
basis  far  a  simple,  elegant  protocol  for  hand-offs,  and 
rendezvous  of  tasks  between  different  control  agents. 

Imagine  two  operators,  one  rn  control  of  the 
tele  robot  and  the  other  about  to  take  over  is  relief  of 
the  first.  Each  operator  would  be  in  control  of  a 
Simula  non  of  the  the  telerobot,  bat  only  the  controls 
signals  of  the  first  would  be  sent  to  the  real  telerobot 
The  relief  operator  would,  with  position  c  hitch 
disengaged,  guide  his/her  simulation  as  dose  to  the 
first  operator's  as  possible  (or  as  dose  as  required,  as 
a  function  of  the  interpolation  and  smoothing 
methods  to  be  used  in  the  rendezvous!.  The  first 
operator  then  disengages  their  position  c latch, 
leaving  the  path  "hanging".  Figure  6  shows  this 
mi-im#wy  in  the  interaction. 

The  second  operator  then  engages  their  position 
clutch,  rendezvousing  with  the  path  and  taking 
control  of  future  path  generation.  When  the  acmal 
manipulator  passes  aver  this  path  segment,  it  will  do 
so  smoothly  and  will  not  notice  that  a  change  of 
control  agent  has  occurred  in  nud-manuevcT.  We 
can  again  find  interesting  biological  analogies  to  this 
visualization  situation.  For  example,  consider  the 
interactions  among  basketball  players  as  they 
previsualize  fast-paced  multiplayer  interactions. 

We  believe  that  this  simple  protocol  can  be  built 
upon  to  mechanize  quite  a  wide  range  of 
manipulation  interactions  between  autonomons 
agents.  See  for  example  the  discussions  in  [CON 87] 
concerning  the  various  scenarios  of  "student  pilot  and 
instructor  pilot',  where  we  consider  han doffs  of 
manipulation  and  cognition  tasks  among  humans, 
between  humans  and  machines,  and  among  machines. 


Figure  6l  Using  dme  and  position  dutches  to 
handnff  task  to  another  forward  simulation  agent. 


INITIAL  EXPERIMENTS  AND  RESULTS 

The  manipulation  tasks  in  oar  initial  experimental 
trials  consoled  c#  having  the  telerobot  touch  a  series 
of  boxes  of  size  S,  each  separated  by  a  distance  D.  By 
vary  ing  D  and  S  we  prodneed  tasks  over  a  range  of 
difficulties-  Our  goals  were  to  test  our  hypotheses 
that  the  lane  and  position  dutches  can  improve  the 
overall  manipulation  performance,  and  also  to  study 
the  detailed  functional  relationship  of  task  time  vs 
task  difficulty  over  a  range  of  system  parameter 
values.  Analyses  of  such  relationships  may  lead  to 
principles  for  the  design  of  future  tele-autonomous 
manipulator  systems. 

We  can  take  a  closer  look  at  our  experimental 
setup  in  the  video-report.  You'll  notice  the  solid 
robot  image  on  the  Silicon  Graphics  IRIS  display 
corresponds  to  the  real  robot.  To  simulate  the  effects 
of  communication  delays,  we  insert  a  time  delay  in 
transmissions  of  information  between  the  user 
control  and  the  real  telerobot.  The  user  controls  the 
tele  robot  through  a  joystick.  The  time  and  position 
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dutches  and  the  brake  are  toot  pedals  on  the  floor. 
While  either  of  the  clutches  is  depressed,  the 
corresponding  synchrony  is  disengaged,  and  while 
the  brake  is  depressed,  the  simulation  backs  up  in 
tune  toward  the  real  telerobot's  position. 

Each  square  block  to  be  touched  is  placed  a  fixed 
distance  D  from  its  predecessor,  but  at  a  randomly 
determined  angle.  The  tests  were  performed  across  a 
a  number  of  subjects,  each  performing  each  test  a 
number  of  times  with  different  random  placements  of 
the  blocks.  Results  were  first  obtained  with  and 
without  time  delays  for  direct  tele-manipulation 
t with  no  forward  simulation),  providing  baselines 
for  the  remaining  trials  of  the  different  modes  of 
operation  over  a  range  of  system  parameters.  The 
principle  system  parameters  varied  during  these 
trials  were: 

(i)  Difficulty  ratio  D/S, 

( ii )  Manipulation  size  scale  D, 

(iii)  Communication  delay  time, 

(iv)  Mode  of  operation  (with/wo  clutches), 

( v >  Robot  joint  angular  velocity  limits. 

The  resulting  data  on  operator  task  specification 
times  and'robotic  manipulation  times  are  summarized 
graphically' in  figures  5.2  and  5.3  in  (CON87).  In 
those  figures  we  see  that  the  times  for  manipulation  in 
presence  of  delays  are  substantially  reduced  by 
forward  simulation  and  then  again  substantially 
reduced  by  use  of  the  new  tele-automation  controls. 
(Substantially  means  a  time  improvement  of  at  least  a 
factor  of  two).  For  some  parameter  values  the  use  of 
the  time  clutch  enabled  operators  to  move  out  far 
ahead  of  the  following  telerobot.  The  use  of  the 
position  clutch  enabled-  operators  to  produce  shorter 
manipulation  path  lengths  on  complex  tasks. 
Demonstrations  of  these  trials  and  also  of  handoffs 
using  the;  position  clutch  are  shown  in  the 
video-report.  Refer  to  [CQN871  for  a  detailed 
presentation  and  analysis  of  our  experimental  results. 

RESEARCH  PLANS  AND  ISSUES 

We  are  augmenting  our  experimental 
environment  to  enable  trials  using  additional  forms 
of  telerobotic  manipulators  and  manipulation 
interface  controls.  We  are  further  analyzing  the 
sensory-cognitive-motor  dynamics  of  the 
human  -machine  combination  in  efforts  to  generate 
additional  testable  hypotheses  regarding  factors 
affecting  performance.  We  are  also  planning  trials  of 
simple  autonomous  activity,  with  the  manipulations 
and  handoffs  being  done  under  the  control  of  AI 
planning  programs. 

This  early  tele-automation  work  suggests  many 
opportunities  for  new  interdisciplinary  interactions 


among  those  interested  in  human-computer 
interaction,  robotics,  and  artificial  intelligence 

There  are  also  challenges  and  opportunities 
concerning  provision  of  appropriate  experimental 
environments  for  such  work  in  the  larger  research 
community.  The  provision  of  shared  access  to 
remote  telemampulation  facilities  might  enable  more 
researchers  to  collaborate  on  the  evolution  of  such 
new  technologies  [CON87). 

SUMMARY 

We  have  begun  to  explore  the  integration  of 
telemampulation  and  autonomous  system  technology. 
We  have  created  new  concepts  that  enable  substantial 
telemanipulation  task  performance  improvements. 
These  improvements  are  applicable  to  manipulations 
controlled  by  humans  or  by  machines.  Our  initial 
hypotheses  concerning  performance  improvements 
are  well  supported  by  our  initial  experiments.  We 
have  also  demonstrated  that  the  new  concepts  also 
enable  simple,  elegant  protocols  for  handoffs  of 
manipulation  tasks  between  autonomous  agents.  We 
believe  that  a  number  of  useful  new  space  and  defense 
applications  can  be  based  upon  these  concepts. 
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abstract 

As  a  result  of  recent  advances  in  artifici"1  intelligence, 
human  cognitive  modelling,  autonomous  systems  and 
xlerobotics.  there  is  an  opportunity  to  broaden  our  concepts  of 
•echnology  for  projecting  action  at  a  distance.  We  draw  on 
these  advances  to  develop  a  conceptual  and  architectural 
framework  that  enables  efficient  projection  in  ame  and  space  c  * 
intermingled  manipulation  and  cognition  tasks. 

Where  Al-based  autonomous  systems  have  previously 
been  concerned  with  human  supervisory  intervention  primarily 
at  a  cognitive  level,  we  add  methods  for  rendezvous,  capture 
and  rehandoff  of  embedded  manipulation  tasks.  Where 
relerobotics  has  been  concerned  with  the  projection  of 
xnsory- motor  manipulation,  we  add  the  projection  of  cognitive 
processing.  Thus  extended,  the  two  technologies  mirror  one 
another  and  merge  into  one  of  'tele-autonomous  systems". 

We  introduce  notions  of  how  the  sensory,  cognitive  and 
motor  funcuons  of  tele-autonomous  systems  can  be  factored 
and  transferred  back  and  forth  between  human  and  machine. 
We  illustrate  how  the  times  to  complete  tele-autonomous  tasks 
can  be  reduced  through  nme  and  space  constraint  relaxations 
effected  through  simple  controls:  We  employ  the  concepts  of 
forward  simulation  and  predictor  display,  augmented  by  "time 
and  posiuon  clutches ",  "time  raoo  controls'"  and  "time  brakes ", 
to  control  the  resulting  manipulation  paths  and  event 
transitions.  We  sketch  some  generic  architectural  and  human 
interface  implications  of  these  methods.  Finally,  we  describe 
our  environment  for  exploring  these  methods  and  the  results  of 
some  recent  experiments. 


1  INTRODUCTION 

In  this  paper  we  draw  on  recent  research  advances  in 
autonomous  systems  and  telerobotics,  and  develop  a 
framework  for  an  integrated  technology  that  enables  efficient 
orojecnon  in  time  and  space  of  intermingled  manipulative  and 
cognitive  tasks.  The  technology  builds  a  bridge  between 
xleroboocs  and  intelligent  autonomous  systems  by  providing 
methods  for  controlling  real-time  transitions  between  human 
and  machine  control  of  remote  events. 

In  this  introductory  section  we  reflect  on  the  paradigms  of 
he  telerobottcs  and  autonomous  systems  research 
communities.  We  illustrate  gaps  in  the  two  paradigms,  and 
apponumoes  for  technology  integration,  by  describing  practical 
asxs  humans  can  do  that  would  be  hard  to  implement  with 
either  technology  alone. 


In  the  later  sections  of  the  paper  we  introduce  functional 
concepts  and  terminology  for  "tele-autonomous"  or 
tele-automation''  technology.  We  present  architectural  and 
control  methods  for  implementing  these  concepts,  discuss  our 
experimental  environment  for  exploring  these  concepts,  and 
finally  present  the  results  of  some  recent  experiments. 

11  Paradigms  of  Teleroboocs 

Up  till  now  the  concerns  of  the  teleroboncs  community- 
have  been  primarily  those  of  the  roboticist  and  control  theorist, 
dealing  with  sensing  the  physical  environment,  measuring 
positions,  forces,  and  accelerations,  and  responding  with 
movements  and  forces  to  directly  manipulate  the  physical 
environment.  The  human  provides  the  cognitive  power  of  the 
system,  with  the  human's  sensory-motor  processing 
intermediated  and  projected  at  a  distance  by  the  machine. 

The  coin  of  the  telerobotics  realm  is  manipulation. 
Support  for  teleroboncs  has  come  primarily  from  DOE  for 
projection  of  manipulation  capability  into  hazardous 
environments.  Support  has  also  come  from  NASA  for 
projection  of  manipulation  into  the  space  environment,  and 
from  DoD  for  undersea  applications.  When  teleroboncists 
discuss  the  projection  of  "autonomous  intelligence "  to  remote 
mechanisms,  the  projected  capabilines  are  usually  envisioned 
as  programs  that  can  be  invoked  to  independently  carry  out 
physical  manipulation  tasks  while  the  human  remains  in  contact 
and  control  at  a  supervisory  level  [SHE86]. 

A  common  goal  of  teleroboocs  research  is  the  producoon 
of  as  realistic  a  sense  of  remote  telepresence  and  telecontrol  for 
the  human  operator  as  possible,  given  physical  constraints  such 
as  communication  delay  times  [NOY84,  SHE861.  The  goal  is 
to  enable  operators  to  do  as  nearly  as  well  at  manipulation  tasks 
as  they  could  do  if  physically  present  at  the  remote  location 

1.2  Paradigms  of  Autonomous  Systems 

Within  the  past  few  years,  the  U.  S.  Department  of 
Defense  has  been  supporting  a  rapidly  growing  autonomous 
syste  ns  research  community  [DAR83,  DAV85].  This 
community's  concerns  are  those  of  computer  scientists  and 
artificial  intelligence  researchers  working  to  produce  self 
contained,  mobile  platforms,  such  ar  the  Autonomous  Land 
Vehicle  (ALV)  [MAR86]  and  various  autonomous  undersea 
vehicles,  that  can  maneuver  around  and  enr-  iy  machine 
cognition  to  seek  high-level  goals  in  their envir .aments.  The 
focus  is  on  mechanization  of  sufficient  cognitive  power  to 
achieve  interesting  goals,  such  as  complex  route  planning  and 
replanning  to  effect  reconnaisance  or  force  projection  missions. 
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and  on  providing  sufficient  perception  and  maneuvering 
capability  to  do  things  like  follow  roads,  avoid  obstacles,  and 
find  things  in  the  environment  IDAR83,  MAR861. 

Given  present  limitations  and  computational  complexity  of 
self-contained  machine  perception  (such  as  machine  vision!, 
thv  sensorv-motor  aspects  of  autonomous  systems  technology 
are  currently  rather  crude  when  compared  to  teleroboncs. 
which  can  exploit  human  perception.  The  community  thus 
tends  to  focus  on  widening  the  exploitation  of  machine 
cognition  on  tasks  that  are  feasible  given  the  envelope  of 
available  perception  technology,  conducting  a  parade!  effort  on 
incremental  enhancement  of  percepnon  technology  performance 
i M ARS6J.  The  coin  of  this  realm  is  cognition,  and  cognitive 
interaction  with  the  environment  at  a  symbolic  level. 

A  common  goal  of  autonomous  systems  research  is  the 
mechanization  of  cognition  and  the  associated  task-dependent 
knowledge  systems  so  that  the  remote  machine  is  as  smart, 
robust,  knowledgeable  and  persistent  as  a  human  might  be  tn 
attempting  to  carry  ou’  its  mission.  Since  the  focus  of  the  work 
is  on  autonomy,  human  supervision  or  interaction  is  seldom 
stressed.  When  the  nonon  of  supervisory  control  appears  tn 
3'itonomous  systems,  it  usually  is  concerned  with  having  the 
human  intervene  if  the  system  ts  'not  smart  enough''  to 
cognitively  handle  a  given  situation  [MAR86], 


I  3  Illustrative  task  examples. 

The  following  task  examples  shed  light  on  our  problem 
space,  and  suggest  opportunities  and  methods  for  blending 
teleroboncs  and  autonomous  systems.  Consider  text  editing  on 
a  workstation.  A  human  operator  can  often  envision  and 
generate  the  command  sequence  to  achieve  a  local  goal  much 
faster  than  the  workstation  can  effect  the  screen  manipulation. 
Thus  the  human  may  quickly  "type  or  mouse  ahead"  (assuming 
the  control  stream  can  be  buffered),  then  shift  their  cogrunve  or 
mampulanon  artennon  to  the  task  to  be  done  when  the  machine 
catches  up  (CAR83).  In  contrast,  ji  teleoperanon  system  the 
operator  is  often  "slaved  tn  real-time''  to  the  local  and  remote 
sensory-motor  apparatus.  Can  we  imagine  a  telerobonc 
analogy  to  type -ahead? 

An  extension  of  "type  anead"  ccurs  when  the  operator 
has  constructed  their  own,  perhaps  'ntelligem.  high-level 
commands  such  as  son  this  list",  or  "send  a  message  to  X  to 
get  the  address  of  Y".  The  operator  may  then  type  ahead  at  a 
ratheT  high  level,  with  each  command  in  the  sequence 
performing  not  just  physical  manipulations  but  also  elaborate 
symbolic  manipulations  of  the  environment  to  eventually 
produce  the  text.  Again,  can  we  imagine  a  telerobouc  analogy? 

Or,  imagine  that  you  are  learning  to  fly  in  aircraft  that  has 
dual  controls.  On  a  given  flight  your  cognition  may  be  just 
fine,  but  you  suddenly  fail  to  manage  a  manipulation  task,  and 
the  instru  .’.or  takes  over.  By  analogy,  an  autonomous  system 
might  be  doing  fine  in  its  cognitive  tasks,  but  might  rt-ed 
occasional  human  help  in  its  "lower-level”  manipulation  tasks. 
For  example,  an  ALV  might  run  off  the  road  and  get  stuck,  and 
require  a  skilled  "telednver"  to  free  it;  this  is  quite  a  different 
form  of  intervention  than  the  "mental"  supervisory  intervention 
usually  envisioned  by  ALV'ers. 

Intervention  inn  an  ongoing  autonomous  manipulation 
task  may  not  be  easy,  since  taking  over  in  mid-maneuver  may 
involve  smoothly  effecting  a  multidimensional  control 
rendezvous.  You  can  study  a  simple  form  of  this  situation  by 
u.wracttng  with  the  cruise  octroi  of  your  automobile 


The  duai-controlled  aircraft  story  yields  several  scenarios 
that  have  interesting  autonomous  system  analogues.  The 
instructor  can  coach  the  student  on  various  senary  -notor 
manipulation  tasks,  and  on  various  cognitive  tasks  such  as 
interpreting  instrument  readings.  Visualize  the  piloung  coach 
as  a  human  supervisor,  and  the  student  as  a  remote 
autonomous  system:  The  coach  can  take  over  either  cogninon 
'correcting  an  instrumentation  interpetanon)  or  take  over 
lower-level  manipulation  (prevent  an  unwanted  stall).  There  ls 
a  matrix  of  possible  division  of  responsibilities.  Sensing, 
thinking,  and  acting  can  be  separately  assigned  and  reassigned 
at  any  moment  to  either  the  supervisor  or  the  system. 

But  what  are  the  embedded  protocols  that  make  such 
human  practices  feasible?  Whai  shared  knowledge  is  involved'1 
1  low  are  the  transitions  performed?  How  do  both  student  and 
coach  know  who  s  doing  whai  at  any  moment?  To  make  the 
pteture  even  more  interesting,  consider  the  fact  that  the  overall 
system  has  full  duality:  the  role  of  coach  and  student  is 
reversible  under  some  situations.  Could  such  insights  have 
architectural  implications  for  general  autonomous  systems’ 

1.4.  Merging  the  augmented  paradigms 

Can  we  somehow  build  a  solid  bridge  between  these  two 
technologies  so  as  to  merge  them1  We  believe  the  answer  is 
yes.  as  discussed  in  following  sections.  We  also  suggest  that 
qualitatively  new  kinds  of  functions  and  new  opportunities  for 
performance  improvement  appear  as  a  result. 


2.  BASIC  TELE-AUTONOMOUS  SYSTEM  CONCEPTS 

Consider  either  a  teleoperator  or  a  supervised  autonomous 
system  consisting  of  (i)  a  human,  and  (ii)  a  machine  that  is 
partly  local  to  the  human  and  partly  at  the  remote  site  of 
intended  projected  activity.  Among  the  concerns  of  architects 
of  such  svsrems  are:  In  specific  sit  ..dons,  what  is  the  human 
best  at?  Worst  at?  What  is  the  machine  best  at?  Worst  at1  How 
can  we  shift  control  between  human  and  machine  to  exploit 
these  capabilities?  Are  there  generic  architectural  principles  to 
draw  on?  What  are  the  constraints  on  ultimate  performance? 
What  traditional  constraints  can  we  find  ways  around?  In  this 
section  we  will  explore  these  quesnons  to  develop  step-by-step 
some  basic  funcnonal  concepts  for  tele-autonomous  systems. 

2.1.  Expanding  the  funenons  of  teleroboncs 

Let  s  first  reexamine  some  of  the  architect  s  questions 
from  the  "tele-autonomous"  point  of  view.  One  important 
onstraim  on  performance  m  certain  key  applications  is  the  none 
delay  for  communications  between  the  local  and  remote 
system.  Noyes  and  Sheridan  [NOY84]  have  innovated  and 
demonstrated  a  very  novel  way  to  cope  with  such  a  delay,  by 
using  a  locally  situated  forward- in-tune  telerobot  simulator,  and 
a  graphical  "predictor  display"  overlay  of  the  forward 
simulation  onto  the  fixe  I-delay  return  video  of  remote 
teleoperanon  (Fig.  2.1).  Such  forward  siroulanon  enables  an 
operator  to  move  the  controls  and  immediately  visualize  the 
effect  of  control  action  without  waiting  for  the  return  video. 
Their  experiments  show  that  the  time  to  perform  mampulanon 
tasks  in  the  presence  of  communicancm  delays  can  be  reduced 
by  exploiting  suen  predictor  displays  (Fig.  2.2). 

But  instead  of  just  finding  ways  to  better  cope  with 
constraints,  can  we  also  find  ways  to  relax  some  constraints? 
Suppose  we  had  a  forward  simulator  and  predictor  display,  but 
were  not  operating  through  a  Large  nme  delay.  Although  we 
needn  t  enter  commands  prior  to  the  observed  time  of  then 
remote  execution,  we  soli  might  want  to  do  so,  and  we  could 


r.gure  2.1  Using  forward  simulation  and  predictor  display 
■.j  cope  with  time  delay  i  after  Noyes  and  Shendan). 

use  a  modified  version  of  forward  simulation  to  do  so.  For 
example,  we  might  be  able  to  enter  commands  a  iot  faster  than 
the  telerobot  could  carry  them  out,  as  in  "type  ahead" 
Graphical  overlay  of  accelerated  forward  simulation  enables  us 
to  do  this,  and  to  then  manage  the  cognitive  limitations  of  short 
term  memory  when  commanding  tn  advance  of  observed 
effects.  Such  simulation  can  be  augmented  by  including 
kinematic  ancVor  dynamic  forwand  simulation  of  portions  of  the 
environment.  The  •  :sult  is  a  sort  of  coordinated  "faster  thar 
real-time  recording,  and  then  real-tune  playback"  form  of 
manipuiancn  control. 

There  appear  to  be  a  number  of  ways  that  displays  of 
nmulanon  overlain  onto  telerobot  video  can  relax  telerooooc 
time-synchrony  cor straints,  yielding  possibilities  for 
improvements  in  manipulation  time  performance.  We  also 
hypothesize  that  freeing  the  operator  from  the  "ttm-slaving' 
and  arxnnon  constraining  aspects  of  amt-synchro ruzed  control 
may  make  a  qualitanve  difference  in  the  subjective  "feel  of  the 
controls"  of  such  systems,  making  them  more  like  the 
controlling  of  one  s  own  limbs.  This  hypothesis  may 
eventually  be  made  testable  by  defining  new  measurements  of 
performance  and  fangue  in  new  forms  of  tclerobotic  situations. 

We  can  also  visuaiize  tclerobotic  manipulation  as 
analogous  to  text  ediang,  in  that  it  is  a  senes  of  sensory-motor 
limited  ULjis  intermixed  with  cogmovely-lirmted  tasks.  Thus  in 
>ome  situations  we  may  oe  able  to  project  intelligent  cognitive 
functions  into  the  manipulation  world  (analogous  to  the  "go 
find  the  address  '  command  during  text  editing)  while 
continuing  direct  editing  manipulations.  Methods  that  relax  the 
constraint  of  command  to-manipuLanon  erne -synchrony  might 
enable  operators  to  better  intermix  such  tasks. 

2  2  Expanding  the  functions  of  autonomous  systems 

The  transfer  of  cognitive  tasks  between  supervising 
human  arr*  remote  machine  is  already  a  part  of  tne  autonomous 
systems  paradigm,  being  based  on  past  artificial  intelligence 
work  on  human  machine  cooperation  in  areas  like  diagnostics, 
design,  advising  and  coaching  [RAY83],  Shifts  between 
machine  and  human  manipulaaon  while  the  machine  retains 
cognitive  control  have  not  usually  been  considered.  However, 
:hese  can  now  be  seen  as  just  a  mirror  image,  role-reversed 
version  of  the  augn.  'ted  telerobotics  described  above  Any 
forward  simulation,  time  manipulation  and  control  methods  that 
work  there  will  apply  here  also.  In  both  cases  we  must  deal 
with  control  and  iiuman  interfacing  of  rendezvous,  carture  and 
rehandoff  of  manipulation  casks  between  human  and  machine. 


Figure  2.2:  Manipulation  time  as  a  function  of 
delay,  Tc.  [HAS86.  NOY84,  SHE86] 

As  we  discover  control  and  human  interface  methods  for 
such  manipulation  task  transfers,  perhaps  we  can  also  gain 
insights  into  how  to  better  structure  the  methods  for  cogmuve 
task  transfer  between  human  supervisor  and  autonorr.  us 
system.  Those  methods  are  presently  rather  ad-hoc.  being 
based  on  diverse  applications  experiences  in  AL  Finally,  there 
is  the  human  interface  challenge  of  presenting  "who  has 
control,  of  whai,  and  at  what  tune  and  position  ?"  The  hur  jtn 
may  set  goals  into  the  autonomous  system,  and  then  later  be 
called  on  to  enter  tasks  to  nelp  the  system  reach  either  cognitive 
or  manipulation  subgoals.  Can  we  use  some  sort  of  task  Lamce 
or  tree,  to  represent  and  interface  the  distributed  tasks 
underway  towards  goals  and  subgoals?  New  human  computer 
interaction  knowledge  and  technology  must  be  developed  to 
support  these  new  possibilities  for  autonomous  systems. 

2.3.  ’ntegtahon  of  tele  autonomous  function. 

The  distinctions  between  tclcrobotics  and  autonomous 
systems  blur  when  the  technologies  are  each  expanoed  as 
discussed  above.  But  we  don't  just  get  the  sum  of  the  two 
technologies.  We  get  a  technology  with  some  new  dimensions 
for  enabling  action  at  a  distance.  This  leads  us  to  wonder  if 
we  should  revise  the  goal  of  teleroboocs.  Could  it  be  possible 
to  project  manipulation  capabilities  to  a  distance  that  arc  better 
in  time  performance  than  those  of  the  unaided  human1 
Considerable  research  will  be  required  to  generate  and  test 
hypotheses  to  determine  feasible  performance  improvements 
ard  limits  of  such  an  exrcided  tele- mam pulanon  technology. 
In  addition  to  examining  new  aspects  of  robotics  and  control 
methods,  this  research  will  also  enter  previously  unexplored 
areas  in  the  psychology  of  human-computer  intcrfWing 


3.  TELE- AUTONOMOUS  SYSTEM  ARCHITECTURE 
AND  CONTROL  METHODS. 

In  this  section  we  describe  methods  for  relaxing  the 
operator  to  manipulator  time- synchrony  constraints  usually 
found  in  telerobotics.  The  first  key  idea  is  the  use  of  a 
tune-clutch'  to  enable  disengagement  of  time  synchrony 
during  path  planning.  We  extend  this  idea  by  adding  a 
’position  clutch*  that  allows  forward  simulation  manipulation 
and  positioning  trials  without  generating  path  plans.  We 
include  a  "hme-ratic  control',  to  enable  variations  in  the  ratio  of 
simulation  time  to  real  time.  We  introduce  the  concep*  of  a 
"time  brake'  to  allow  the  forward  simulation  to  be  'braked' 
bark  in  tune  to  avoid  unfi.seen  contingencies.  We  then 


provide  scenarios  of  how  these  new  controls  might  enabte 
operators  of  the  augmented  systems  to  achieve  considerable 
nme  improvements  in  certain  manipulation  tasks.  We  also 
suggest  how  the  augmented  architectures  enable  easy 
transitions  of  control  of  cognition  and  manipulation  tasks 
between  human  and  machine,  thus  enabling  integrations  and 
mirrorings  of  telerobonc  and  intelligent  autonomous  functions. 

3. !  Disengaging  ame -control  synchrony  using  a  Time -Clutch 

We  bnld  upon  the  forward  simulation  and  predictor 
display  concept  of  Noyes  and  Sheridan  as  follows.  Suppose 
we  are  using  a  telerobonc  system  as  in  Fig.  2. 1 .  We  :  ugment 
the  system  with  a  control  that  we  call  a  "Time  Clutch".  This 
control  enables  us  to  disengage  the  'direct  gearing'-  or  time -rate 
■  but  not  absolute  time)  synchrony  of  simulated  time  and 
real-nme,  ana  move  the  forward  simulator  ahead  as  fast  as  skill 
and  ]udgement  will  allow.  The  predictor  display  presents  a 
forward  path  as  a  goal,  tlut  is  as  a  sequence  of  point  positions 
to  be  followed  by  the  system  as  fast  as  is  feasible.  Note  that 
the  path  could  be  generated  subject  to  some  settable  mean  error 
parameter,  for  example  as  a  "tube''  of  given  radius  (SUH87J. 

The  nme  clutch  enables  an  operator  to  disengage  from 
real-nme,  and  manipulate  ahead  of  the  displayed  video  of  the 
real  manipulator  by  working  the  overlay  wire-frame  figure  of 
the  manipulator  on  the  predictor  display.  Example  situations 
where  tins  would  have  benefit  would  be  dunng  slow 
movements  of  large  space  structures  and  in  slow  undersea 
vehicle  manipulations.  The  operator  can  thus  do  the  telerobonc 
equivalent  of  rype-ahead"  and  then  perhaps  slow  down  and 
carefully  posinon  for  some  tricky  maneuver.  We  hypothesize 
that  tn  many  manipulation  task  sequences  such  time  saving 
accumulations  and  later  exploitations  will  be  possible,  thus 
reducing  overa!1  manipulation  task  times  and  a)so  the  fraction 
of  the  task  nme  that  requires  one rator  involvement. 

How  can  we  implement  the  time  clutch  control?  How  is 
the  system  to  determine  the  path  as  a  function  of  tune  when  the 
dutch  is  disengaged1  The  tune  clutch  can  be  thought  of  as  a 
simple  switch  used  to  make  or  break  the  connections  within  the 
kinemanc/dynamic  robot  simulation  that  would  normally 
constrain  the  rate  at  which  the  forward  simulator  could  be 


slewed  around  in  space.  When  the  dutch  .s  engaged.  :ne 
position  (or  rate)  joysnek  control  of  the  simulator  is  sampled 
and  directly  controls  the  movements  of  a  amuiator  model 
which  is  constrained  m  its  movement  rates  and  accelerations  as 
if  it  were  a  real  robot.  A  buffer  is  inserted  between  -.he 
simulator  and  the  telerobonc  manipulator,  to  hold  the  stream  of 
sampled  position  increments  as  incremental  move  :o 
commands  (see  Fig.  3.1). 

With  the  nme  clutch  is  engaged,  the  command  buffer 
presents  a  stream  of  posinon  points  at  a  fixed  sample  rate,  and 
the  telerobot  can  simply  increment  its  position  accordingly 
But  when  the  time  clutch  is  disengaged,  the  dtstance  between 
successive  path  posmons  may  be  greater  than  the  telerobot  can 
move  tn  a  ame  sample,  and  an  interpolator  is  used  to  generate 
intermediate  points  along  the  path.  This  interpolator  can 
always  be  active,  witn  the  only  difference  in  function  upon 
disengaging  the  ame  clutch  being  the  breaking  of  simulator 
contraints  on  simulator  velociaei  and  acceleranons.  In 
sophisticated  systems  where  telemampuiatw  touch  sensing  and 
force-sensing  dunng  interacnons  with  the  environment  are 
reflected  back  to  the  manipulator  operator,  disengagement  of 
the  ame  clutch  must  also  disengage  these  reflected  forces  and 
subsutute  simulated  forces  generated  by  the  simulator 

3.2  Disengaging  position  synchrony  using  a  Fosinon  Clutch 

In  some  cases,  we  may  want  to  move  the  forward 
simulator  in  space  without  actually  sampling  the  path,  for 
example  to  pre-posmon  for  a  complex  mampuianon.  Thus  we 
may  wish  to  disengage  the  simulator  from  recording  any 
positioning  commands.  To  do  this  w:  disengage  a 
Position-Clutch"  mat  allows  forward  simulaaon  without  path 
planning.  This  provides  a  positioning-synchrony  constraint 
relaxation  analogous  to  the  earlier  time-synchrony  relaxaoon 
In  this  case  no  position  information  is  entered  into  the 
command  buffer  ur.nl  the  position  clutch  is  reengaged,  at  which 
ame  the  reengagement  position  is  entered  into  ihe  control 
buffer,  and  later  used  by  the  actual  robot  in  path  interpolation 
from  the  previous  path  position.  If  the  real  system  catches  up 
with  a  position-clutch  disengagement  point,  it  hits  an  empty 
mark"  in  the  command  buffer  and  must  wait  for  funner  path 
data  t«  enter  the  buffer  (see  Fig.  3. 1 ). 


Figure  3.1:  Using  time  and 
posinon  clutches,  tinv*  rauo  and 
nme  brake  to  coiurot  forward 
simulation  path  planning  in  a 
tele-autonomous  system. 


Note  that  the  urne-ctuurh  can  be  disengaged  while  the 
pnsmon  clutch  is  engaged.  But  disengaging  the  position  vlutch 
overrides  any  actions  of  the  time  clutch.  Reengaging  the 
time-clutch  after  an  interval  of  nme-saving  places  the  forward 
emulator  and  predictor  display  in  much  the  same  relationship 
to  the  remote  unit  as  when  operating  through  a  time  delay,  with 
the  operator  directly  generating  a  synchronized  'irr.e  and 
position  trajectorv  in  advance  of  the  return  video.  In  all  these 
rases,  use  of  time  ana  position  clutches  can  be  superimposed 
net  time  delavs  in  the  communications  between  the  local  and 
remote  machines  :  assuming  adequate  buffer  capacity.!. 

The  command  buffer  can  be  constructed  to  hold  more 
complex  commands  lin  parallel)  than  just  simple  moves, 
enabling  the  operator  to  marie  certain  path  positions  as  places 
where  an  embedded  task  is  to  be  done.  For  example,  suppose  a 
i  witch  must  be  pjshed  at  some  oomt  along  a  pain,  and  that  the 
manipulation  program  for  switch  pushing  resides  in  the  remote 
controller.  The  operator  might  just  mark  the  spot  on  the  path 
when  the  forward  simulator  reached  the  switch  (momentarily 
disengaging  the  nme  and  position  clutches  and  manipulating  a 
screen  menu  entry  signifying  switch  pushing).  The  telerobot 
manipulator  (or  remote  vehicle,  etc.)  would  then  execute  the 
task  when  it  arrived  at  that  point  on  the  real  path,  ix..  when 
that  path  information  emerged  from  the  command  buffer. 

3  3  Scenario  showing  use  of  the  Time  and  Position  Gutches 

A  short  scenario  for  using  the  two  clutches  follows:  We 
perform  a  complex  maneuver  with  clutches  engaged.  We  then 
disengage  the  ome-clntch  to  quickly  hop  over  a  srmes  of  simple 
manipulation  movements,  such  as  pushing  a  senes  of  switches 
A  faint  "smoketmT  supenmposes  the  forward  simulation  path 
over  the  return  video  display,  helping  us  visualize  our  progress 
along  the  chosen  path.  Having  saved  some  nme.  we  then 
disengage  the  position  clutch,  and  by  tnai  and  error  movements 
position  ocr  manipulator" in  simulaoon  to  be  at  the  right  place  to 
begin  a  complex  maneuver.  During  this  phase,  the 
suoulanon-generatexf  manipulator  image  moves  on  the  display, 
bdt  leaves  no  smoketraiP  of  a  committed  path.  Upon 
reaching  the  correct  position  and  orientation  to  begin  the  nex' 
maneuver,  we  reengage  both  clutches  (the  "smoketrai!  will  now 
display  the  new  interpolated  path  segment)  and  wait  for  the 
remote  system  to  catch  up  We  then  begut  the  maneuver.  In 
■his  way  we  (i)  save  some  time,  fij)  use  the  time  saved  to  later 
preposition  for  another  action,  (iu)  avoid  taking  the  actual 
remote  system  through-  complex,  manrpularively  unnecessary 
preposioomng  movements,  and  Civ)  do  this  all  in  a  natural  way 
‘  rough  simple  cc  mis 

Note  that  the  following  of  paths  generated  during 
:;me-clutch  disengagements  can  be  done  by  crude  methods 
'Ucb  as  simple  interpolations  while  keeping  movements  slow 
trough  to  avoid  robot  rate  limits.  Or  it  could  be  done  by 
sophisticated  methods  that  take  into  account  the  full  dynamirs 
of  the  situation  and  drive  the  remote  telerobot  at  nearly  us 
maximum  feasible  rate  along  the  path,  given  specific  actuator 
omits  and  desired  mean-error  limits.  This  defines  a  large 
tradeoff  space  in  the  computational  complexity  of  trajectory 
generation  vs  the  time-performance  and  robustness  of  the 
resulting  manipulation. 

5  4  Tune- Rano  Control. 

So  far  we  have  implied  all  ratio  of  forward  simulation 
ame  to  real  tune  when  a  tele-autonomous  system  is  operated  in 
time- synchronized  mode  (time  clutch  engaged).  This  needn't 
be  the  case  Foe  example,  we  might  be  able  to  operate  the 
simulator  much  faster  than  the  telerobot  can  follow,  and  wish 


io  plan  the  path  sequence  in  a  synchronized,  but  waled,  one 
So  instead  of  using  the  time-clutch  to  disengage  time 
v.nchrony,  we  might  want  to  establish  a  low  time  ratio 
oeiween  simulated  and  real  time.  But  there  also  mignt  be  tasks 
that  the  telerobot  can  do  far  more  rapidly  than  we  could 
prescribe  with  the  simulator.  In  those  situations,  if  we  had 
saved  up  '  some  ume.  we  could  establish  a  high  tune-ratio  ot 
simulated  ume  to  real  ume  and  slowly  perform  a  maneuver  to 
be  later  done  very  rapidly  by  the  telerobot  (when  it  catches  up 
:o  that  section  of  the  path). 

These  ume-ratio'"  scalings  relating  real-time  .o  simulated 
time  can  be  easily  implemented  a rxi  then  controlled  bv  allowing 
a  change  of  time-ratio  while  the  tune-clutch  is  disengaged 
(analogous  to  changing  the  gear-rano  of  a  vehicle  while  the 
clutch  is  disergaged).  The  dme-rano  then  holds  us  new  value 
unrii  changed  again  during  a  laier  time -clutch  disengagement. 
Time-rano  scaling  should  not  be  confused  with  operating  while 
the  nme  dutch  is  disengaged  (where  no  fixed  relationship  is 
specified  between  simulator  ume  to  generate  a  path  ini' 
teieroboi  time  to  follow  the  path). 


5  5  Handling  of  contingencies  by  using  Time  Brakes 

What  are  we  to  do  if  we  are  forward  simulaung  wav  out 
:n  front  of  the  teiemanipulator  and  suddenly  see  (in  return 
video)  something  intrude  into  the  planned  path  of  the 
manipulator9  To  handle  such  simple  contingencies,  we 
introduce  a  mechanism  we  call  a  "Time  Brake".  Depression  of 
the  ume  brake  disengages  the  clutches  and  "decelerates 
simulated  time"  by  incrementally  extracting  (LIFO)  previously 
generated  posinon  commands  from  the  command  buffer.  The 
forward  simulator  is  correspondingly  moved  in  reverse  back 
down  the  path.  This  allows  the  operator  to  move  (as  quickly  as 
desired)  back  in  time  along  the  forward  simulation  path  unnl 
located  in  space  on  the  earlier  side  of  the  obstacle.  We  also 
provide  an  emergency  brake"’  that  "immediately"  empues  the 
command  buffer  and  halts  the  telemanipulator  (subject,  of 
course,  to  overshoots  due  to  manipulator  compliance  and/or 
dynamic  constraint  management,  and  to  races  against  1/2  Tc). 

3  6  Manipulation  and  cognition  control -cran si d on s  and  their 
rrurronngs  in  tclerobooc  and  supervised  autonomous  systems 

Tne  control  methods  described  in  this  paper  enable  simple 
and  smooth  handoffs  from  local  human  teleoperation  control  to 
and  from  remote  machine  manipulation  control  (using 
downloaded  manipulation  commands).  But  they  also  provide  a 
base-level  protocol  that  enables  easy  mechanizations  of  the 
other  types  of  transitions  from  local-control  by  human  or 
machine  of  cogniuon-or-manipularion,  to  local-or-remote 
machine  control  of  oumpularioo-of -cognition.  Seen  this  way, 
the  augmented  teleoperanon  and  autonomous  systems  mirror 
into  one  another  to  become  tele-autonomous  systems.  Human 
or  machine  agents  on  'either  side  of  the  mirror"  can  exploit 
similar  forward  simulation  and  control  handoff  methods. 

We  hypothesize  that  human  operators  of  this  technology 
can  ieam  to  accomplish  graceful  and  efficient  hand-offs, 
rendezvous  and  recaptures  of  real-time  thinking  and 
manipulation  basks-,  and  that  human-or-mactaine 
cognidon-or-marupulation  operators  ca*,  also  exploit  the 
forward  simulation  constraint  relaxations  to  improve 
performance  in  many  situations.  Humans  could  thus 
supervise,  or  be  dynamically  embedded  into,  complex 
human-machine  task  lattices,  taking  and  releasing  control  of 
subtasks  at  appropriate  times  anu  places. 


Fat  exwnpte.  the  tdMstaniDUi  technology  pnnda  a 
framework:  that  enable*  u*  ter  rmnne  the  aircraft 
stademHAStraaor  scenario,  with  other  student  or  inamriir 
being;  machine  or  human,  each  undertaking  sequences  of 
cognition  and  manipulation  tasks.  Conn  (ter  for  example  the 
a  manor  in  Figure  1JL,  where  we  sees  telerofwt,  ff,  following 
a  path  specified  by  a  forward.  simuLarian.  S,  tharis-  proceeding 
with  its  time  clutch  disengaged-  The  operator  of  simulator  3 
then  disengages  the  pun  non  clutch  and  moves  3  down  and  to 
tbe  right  S  ls  now  essentially  disen gagxt  from  any  axmecncKr 
with  the  teteroboL  Ar  the  same  time,  some  other  operamr 
(human  cr  artificial)  is  maneuvering  anothar  amulatrat  S\ 
down  towards  the  forward  planned  path  (S'  ls  also  operating 
witlr  its  potman  clutch  disengaged).  When  S’  gets-  "close 
enough"  to  the  ptant  where  S  leff  crfFpaih  planning,  $  can  then 
engage  its  position  clutch,  and  take  over  central  of  the  tderoboc 
(subject  to  acquisition  imermediatKm  by  arbiter  cr  collision; 
detect  mechanism  at  the  teierobn).  The  Lmeractuar  controls  car 
be  factored  from  the  actual  manipulations  illicit .  r  a  small,  but 
important,  time  increment  The  two  "players"  can  formulate 
and  interact  using  shared  visualizations  and  rapid  cueing 
methods,  much  as  skilled  sports  players  leam  to  do.  This 
:  unpie  example  is  suggestive  of  a  number  of  more  elaborate 
prouncois  and  scenarios  dun  can  be  constructed  on  top  of  thr- 
low-level  hand -off  and  rendezvous  protocol . 


Figure  3.2:  Using  tune  and  post  don  clutches  to 
"hand-afT  and  to  "rendezvous"  with  a  mam  political  taste 


4  INITIAL  RESEARCH  ISSUES  AND  HYPOTHESES 

Our  initial  approach  «o  tete-aratmemous  rysaem  research  is- 
to  form  hypotheses  concerning  the  overall  human-machine 
system  touch  as  current  human-computer  interaction  weak, 
models  unassisted  human  prra*pin»l  cognitive;  and  motor 
systems.  We  then  tea  these  ideas  by  experiment  For  example, 
"*'s  law  (CAR83)  predicts  that  the  time  for  the  eye-tmod-harvi 
c> —  of  touching  an  object  of  linear  sire  S  at  a  distance  D  is 
given  by  T  =  Klog2(D/S  +•  0-5),  wboe  t  ~  100  msec -/bit 
Therefore,  ample  tasks  based  on  varying  the  relative  sizm  of 
objects,  and  the  distances  between  objects,  might  produce 
meaningful  trials  of  the  various  nrmttet  of  tele- autonomous- 
operation  Could  performance  operate  under  some  sort  of 
seated  Fitt's  law  in  seme  modes?  Or  is  it  more  -  mptex  thxt 
that?  We  could  find  out,  and  perhaps  deveiuo  v.  ,je  insights 
and  principles  on  herw  tn  best  design  such  system* 


A  ample  7-itimi«nn»l  ipwtwit  mr  «, ,  i snodate  s  wide 
range  of  such  [F»fi*m»nry  rrrel^  writ  tUthe  nwTrjviiarvwr  rrrat 
-kj-irhrri-  in  figure  4L  In  that  Wpmr  w  see  a  numbs  of 
"switches-"  of  tinea:  sze  "S'  located:  tn  yipimor  at  known 
posmrms-  ur  the  manipulator  workspace!  Each  switch  is 
distance  "O'  from  us.  predecessor  at  the  sequence  The 
objective  is  to  much  each  swuch  at  the  unpwnap  as  rapidly  as 
pussible  Increasing  the  rano  D/S  uaieapunds  to  increasing 
task  manipulation  comptexity,  pagwhiy  requiring  naze  nmr-  for 
nun  puLancrr  convergence,  as  attracted  tn  Eirfs  lass. 


We  can;  expLtanr  and!  answer  many  questions,  using  this 
simple  retried  What  is  the  fiincnonal.  fbcrrr  of  the  reduenoo  in 
manipulation  time;  over  direct  trienperaonn-  that  uan  be 
obtained:  when  using  the  diffinmi.  aigmaiumans  of  canoed? 
How  tag  thcMcfiunci  n  sis  antfimag,  affected:  in  the  pieaacc  qf 
aamnmricanars  delays?1  Haw  ate  the  times  affixtetf  by  the 

rtiffvnifty  nf  ifw  rnmmjy>l»nrT  largmfrrg  nwl«  (Tajyr  rm— »r  onliu-» 

of  D/S).  What  ace  the  effects  a£  other  syson  pacameina,  such 
as  joystick  force  constants  and;  roftac  vefoggy  limns?  Wftnr 
tteiemnims  the  percentage  ai  the  ra»k  esesamon  one  tfas  the 
sysnmr  operator  needtft  be  in  the  ctannte  loap^  so  that  they  can 
beavailahtefrrppfiaiiiiiigcahBrfimcnana? 

The  qnantrtmvem-!i»  of  wich  tomfsarr  yield  ini|ima»iT 
early  mrmim  of  the  fnm»  anrf  dimensions  of  performance 
improvements  pmbie  with  the  tete-enanomoua  ctaurols  Tbe 
results  can  then  help  gniite  planning  of  fim+nr-  cnais  »rwt  the 
expkaatnry  evolution  of  the  technology. 

5.  EXPERIMENTAL  ENVIRONMENT  AND  RESULTS 


We  ne  building  a  general  experimental  environment  in 
which  to  u:enc  and  evolve  trie-autonomous  technology.  Cn  this 
section  we  describe  our  initial  facility  amt  some  early 
experiments  with  the  different  control  methods  These  early 
experiments  are  being  dene  on.  transitions  viewed  from  the 
teUsobotie  pourr  of  view,  and  provide  the  basis  for  planning 
later  experiments  in  which  we  will  study  transitions  from  the 
various  mixed  teletobooc/aumnomnus  systems  poino  of  view. 


Our  urinal  facility  consists  of  a  Utmnanan  r’UMA  5fiO 
used  as  a  teterobot.  controlled  by  other  a  DEC  VAX  I  r/730> 
computer  or  an  Apollo  DSP90  computer  (both  modes  are 


available  l  A  farce  ami  pmgnt  amarove  joystick  *  land  it  sn 
input  device  id  provide  a  rate  nrpul  for  the  gtautaoi  To 
-unrmlaic  a  vancry  of  real  robots  ryptcal  of  thooe  urod  in  space 
or  underaea  operations,  apprupnaa*  velocity  Irmas  are  placed 
cm  cads  jam.  To  simulate  rerraaencs  of  the  tamtam  from  the 
operator,  a  variable  delay  can  he  mserttri  between  a  forward 
srtn ula tor  generated  control  stream  and  the  PUMA,  usmp 
Ktrffer  co  hold  the  trajectory  sample  stream. 

A  high- performance  Silicon  Graphics  IRIS  woritsntnon  i 
ived  to  generate  and  mix  the  display  of  da  forward  snnulato 
and  the  teleroboi  with  the  taerobot  seen  either  tn  return  video 
or  as  a  graphics  model  ttfae  latgr  can  he  useful  max  n  is  both 
easier  to  obtain  correspondence  between  the  overlay  and  the 
smmlanoo  and  easier  to  modify  the  vxwptnm  of  the  sysnrn). 

Simple  time  and  pstacoo  clutches  have  hern  implemented 
tn  the  system.  The  logical  operant®  of  the  c  lute  he*  is 
portrayed  as  a  stag  diagram  m  Figmr  5.1.,  which  show*  the 
allowable  combhumons  of  time  and/or  position  synchrony  and 
the  transitions  between  them.  There  ate  three  allowable  control 
states  from  the  taenperanon  ptutrt  erf  went  1)  TSyn  £  ffi-yn  — 
time  and  potation  synchrony.  2)  T^yn  —  posnson  synchnansm 
only,  and  3)  NoSyn  —  both  time  and  position  synchrony 
disengaged.  Changes  between  these  states  axe  nmtmlhd  by 
swuches  that  operauos  push  with  their  tail  —  similar  tn  the 
clutch  m  an  antcanotaita  The  joystick  moves  the  smulmcr  m 
all  three  states,  but  the  sene  determines  the  effect  of  shnulairr 
rrvn— merit  an  path  planning  and  path  buffer  eiaaading. 


Figure  5  1  Diagram  of  forward  sntadaanr  sales  ard 
the  transmons  caused  by  time  sod  fm*™  clutches. 

A  foot  operated  tune  brake  «  rmptancnitd  which  wtm 
pressed  (i)  disengages  both  clutches,  an  ovexnding. 

transition  to  NoSyn,  and  (ii)  begins  detamg  entres  in  the 
command  buffer  (LfFO),  thus  running  the  forward  trrrmlsmr 
back  down  the  previously  generated  path. 

'n  the  time  and  position  synchronized  stair  (TSyn  A 
PSyrw,  the  force  and  moment  outputs  of  the  joystick  at 
sampled  at  the  mput  rate  required  by  ttr  PUMA.  The  fames 
and  moments  obtained  ate  treated  as  vectors  of  n^rrrrsl 
velocities  ui  Cartesian  space.  These  tie  imwgr»»«l  to  obtain 
poamon  samples,  and  these  samples  are  placed  in  the  onmnaivl 
buffer.  While  in  ante  synchronized  mode  the  buffo-  is  emptied 
at  the  same  raze  it  is  filled,  and  the  values  obtained  from  the 
buffer  axe  input  to  the  PUMA  sysaon,  which  neats  each  sample 


as  a  goal  to  reach  m  its  sample  penart,  by  slewing  any  cr  all  of 
us  it*  joints.  When  we  wish  to  simulate  situations  where  the 
wettons  can  be  visualized  and  simulated  much  faster  than  they 
can  he  manipulated,  such  as  when  mowing  large  structures  tn 
space  fNASRl]  or  underwater,  we  place  a  ae leered  jntr-:!ar 
velocity  limit  Wj(i)  <  Wjmaxlu  on  each  aus,  u  of  the  HIW-1 

When  the  time  clutch  is  disengaged,  and  the  podm® 
‘.yncromzcd  (PSyn)  tram  is  er  eted,  the  robot-model  pNy'i1 
constraints  on  simulation  distance  ctwnrd  per  time- sample  axe 
removed,  and  the  iminl  joystick  force  and  torque  constants. are 
multiplied  by  a  gam  constant,  Gs,  enabling  the  operator  to 
rapidly  slew  the  simulator.  Path  samples  may  be  generated  at 
varymgly  wider  paid  mtnsvmls  than  is  possible  when  control  a 
synchronised  in  tune  Valuer  removed  from  the  buffer  may 
this  request  incremental  moves  larger  than  con  be 
accomplished  in  one  PUMA  sample  period,  given  selected 
constraints  on  the  angular  velocities  of  the  PUMA  jomts. 
When  this  occurs,  me  commanded  move  is  interpolated  and 
spread  across  more  than  oac  PUMA  sampling  interval,  with  the 
actual  PUMA  rate  of  motion  constrained  as  above  by  the 
selected  pom  angular  velacny  constraints-  Thus,  the  srmntaied 
tetanboc  can  hung Iran  along  ■  path  swell  ahead  of  thrreal 
tetaohnt,  and  we  can  "saw  up  atone  one"  In  addition,  the 
teal  iotaot  follows  this  path  at  nearly  iss  maximum  rate,  for  a 
given  set  of  valuta  of  Wjnrax  (i).  Thus  we  predict  that  the 
overall  manipumtHm  time.  Tin,  will  usually  be  rmoiler  using 
this  made  than  if  their  were*  Bated  rat®  between  simulation 
tune  and  real- am:  (as  for  example  in  time-rat®  control). 

ff  a  mutation  is  made  from  PSyn  to  the  imsyivhronnxri 
(NoSyn)  soar,  the  operator  becomes  free  to  move  the  nmulaod 
taerobet  without  values  being  placed  in  the  comuauid  buffer. 
Then,  when  a  transtoon  is  made  back  to  PSyn.  the  current 
position  of  the  omntaoH  ttJgohut  is  pheed  rn  the  buffer. 
When  this  value  is  exu acted  from  the  huffer.  the  tetatrbot 
makes  mterpn bread  inczemcmal  moves  directly  towards  ihat 
ittaind  pnsihnn,  without  going  through  all  of  the  mouems  the 
openura’  had  to  use  to  gHto-thstpowiaxi.  This,  enables  'saved 
up*  time  to  he  used  to  "edit”  out  some  teal  tune  and  path 
'ratal. r»  uarri,  fer  example,  fer  oanplex  paeposnaxung. 

5_L  Fa)rjnniraiiaJParanaBar»a.Trwb.aixl  Results 

in  rxii  finu trials  %  nail  img| iA  unban  "  ilim  miisal. 
5  iirti  regfaiis  nualai  ®thaoaFignre4JT  Wecooducteda 
■anes-ofmals 

G)  Three  diftaem  nrfyrrq  (X,  Y^JZ>cach  performed  a  eies 
manipulation  gales  using  ita  aagbed.  Two  times  were 
rwaratal  fer  each  trial:  The  ■abject's  nine  to  specify  the 
mantpalanpo,  (Ts),  xcd  the  qaua’s  tme  ®  couipkn.  the 
montpalarioc  (Tmf.  We  alas  recorded  the  ranual 
pa— pal  atari  path  length,  Los.  The  rat®  of  Lm  to  the 
i  main  watt  path  f-  5D)  pro*  aha  s  measure  of  one 
dmasauno  of  operaatr  skill). 

(ii)  The  series  df  swischnoaching-tatas  varied  than  ample  » 

difficait  by  ranging  fen  low  valaes  trfTlfft  *•»  high  values 
of  DdS  (D  *  500  mmq  S  =  25,  50, 75  rj. 

(iii)  Qxnmnmcadoo  delays,  Tc,  of  0.  *  aa.  -  roc.  act  "»»rt 

(iv)  Tasks  over  the  range  of  difficulty  tzol  the  range  of 
communication  delays  were  perfonnod  by  each  subject 
using:  (s)  direct  teleopersnan  (TOP),  (h)  teknper  tat 
assisted  by  forward  simulation  (TOP+FS),  and  (c) 
teleoperauoo  assisted  by  forward  u  mu  la  non  and  time 
clutching  (TOP+FS+TO. 


During  these  iirst  trials,  other  key  system  parameters  were  heii:  . 
^oostam  as  follows: 

d)  Workspace  to  monitor- screen  length-raao  =  8:1. 

ui)  Joystick  sample  period  =  0  017  sec. 

u.i)  Joystick  force  constant  =  0.01  mm  per  oz  per  sample 
period  =  0.6  mm  per  sec.  per  oz. 

.:v)  Joystick  .^rque  constant  =  0.0012  rad.  per  oz-tn  per  sec. 

iv)  Joystick  gain  constant,  Gs,  m  (TOP+FS+TC)  =  4.0. 

( vi )  Angular  velocities  of  ail  6  PUMA  joints  were  limited  w 
Wj  <  Wjmax  =  0.5  radians  per  sec.  (but  see  also  below). 

Other  comments  on  our  methods:  The  chosen  constant 
values  yield  moderately  responsive  controls  when  moderate 
joystick  forces  and  torques  are  applied.  The  angular  velocity 
limits  yield  a  moderately  fast  robot  (slower  than  the  PUMA  can 
go  at  its  fastest,  but  very,  very  much  faster  than  a  scaled  shuttle 
arm).  All  subjects  engaged  in  preliminary  learning  trials.  Ail 
used  the  joystick  "one- handed".  Trials  began  after  a  period  of 
preliminary  learning.  Comparable  power-law  of  practice 
performance  levels  [CAR83]  were  recorded  for  each  mode. 

Results  of  some  of  these  initial  trials  for  one  subject  are 
plotted  in  Figures  5.2  ,  which  shows  die  specification  dtne  (Ts). 
and  manipulation  times  (Tm)  for  tasks  over  the  range  of  D/S 
difficulty  holding  D  =  500mm.  Included  are  resuhsTor 
communication  delays,  Tc,  of  0.0,  2.0  and  4.0  seconds.  The 
results  are  displayed  for  the  dure  relevant  modalities  of  comroUu 
fa)  TOP.  (b)  (TOP+FS),  and  (c)  (TOP+FS+TC). 

We  note  that  a  comparison  of  TOP  and  (TOP+FS)  repeats 
experiments  of  S  hen  dan,  et.  al.  [HAS86,  SHE86],  confirming 
the  results  of  that  wort  We  see  that  (TOP+FS)  gives  a 
significant  gain  in  both  Ts  and  Tm  over  TOP  alone.  Theh  we 
nonce  that  (TOP+FS+TC)  gives  another  significant  gain  in  1  < 
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over  (TOP+FS).  In  the  initial  trials,  we  found  that  Wjmax  = 

1  0  radvsec.  was  high  enough  for  the  robot  s  Tm  ame  to  keep 
■jp  wuh  even  the  shortest  (TOP+FS+TC)  Ts  ames  (see  Fig 
5  2).  We  then  found  that  Wjmax  =  0.5  rad./sec.  constrained 
Tm  so  that  subjects  could  easily  outpace  the  rooot  and  save  up 
time  (see  Ftg.  5  2).  Many  of  the  initially  hypothesized  forms  of 
results  were  demonstrated  using  these  parameter  ranges. 

We  then  nouced  that  Ts  and  Tm  grew  less  rapidly  in  D/S 
than  anticipated  We  hypothesized  that  D  =  500mm  was  large 
enough,  given  the  joysnek  constants  a«l  Wj  values,  to  produce 
dynamic  constraints  related  to  D  and  not  just  D/S.  So  we 
repeated  scaled  versions  of  these  trials  at  smaller  values  of  D. 

Figure  5.3  shows  the  results  for  D  =  250  mm  and  S  =  50, 
37  5,  25  and  12  5  mm.  (wuh  work  to  screen  scale  =  16: 1.  and 
Wjmax  =  0.5).  It  also  includes  results  using  D  =  125mm  and 
S  =  25,  18.7,  12.5  and  6.2  mm  (with  work  to  screen  scale  = 
32:1.  and  Wjmax  =  0.5).  These  results  are  interesting,  because 
for  all  three  modes  the  data  per  mode  at  D  =  250tnm  and  D  = 
125  mm  essentially  fall  on  top  of  one  another.  The  250mm  and 
125mm  curves  for  each  mode  lie  well  below  those  for  D  = 
500mm.  Refer  Fig.  5.2  for  the  time-clutch  mode  data  for  D  = 
500mm  (it  would  partly  overly  the  Fig.  5.3  time-clutch  data). 

At  this  scale  the  system  operates  m  a  "Fin's  law-like ' 
region,  with  Ts  and  Tm  being  functions  of  D/S  (but  not  D), 
with  the  values  in  most  cases  at  D/S  •=  20  about  twice  those  at 
D/S  -  5.  For  Wjmax  =  0.5,  die  robot's  Tmat  this  scale  could 
stay  up  wuh  the  subjects  Ts.  We  varied  Wjmax  and  found 
values  of  0.35  (for  D  =  250)  and  0.25  (for  D  =  125)  that 
yielded  demos  of  significant  tune  differences  between  Tm  and 
Ts  for  (TOP+FS+TC)  mode  on  the  easier  tasks  (see  Fig.  5.‘ 
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Figure  5  2:  Initial  tna!  results,  showing  Ts,  Trn  as  functions 
of  system  and  task  parameters  for  three  modes  of  control. 


Figure  5.3:  Trials  showing  TS,  Tm  as  ftinctions  of  system 
and  task  parameters,  for  sevsr^  values  of  task  size  scale  D. 


On  further  scaling-down  of  D,  the  system  enters  :t 
Heisenberg'  region  on  the  harder  tasks  ;S  <  2  to  3mm 
Position  sample-sizes,  interpolator  discretization  and  operator 
iters  cause  large  increases  and  variances  in  Tm  and  Tv  i.ik- 
t. mg  to  poke  at  ’Jungs  with  a  needle  under  a  microscope!. 


Throughout  the  trials,  subjects  noticed  striking  difference 
•i  the  tee;  of  the  different  control  modes,  and  dev  elope 
.peciai  tactics  for  coping  with  each  mode.  Most  treated  TOP 
,.-.e  presence  of  delays  like  hitting  a  venes  of  successive!, 
snorter  coif  shots  ',  tr.ing  to  get  closer  each  time.  Subjects 
controlled  (TOP+FS)  aggressively,  firmly  driving  the  simulator 
to  each  swatch.  The  iTOP+FS-TO  mode  was  usually  handled 
with  finesse,  so  as  to  drive  it  fast,  but  not  so  tast  as  to  yield  a 
wild  path  and  thus  large  Tm  and  large  Lrrv5D. 

In  addition  to  they  preliminary  quantitative  results,  we 
have  demonstrated  the  use  of  the  position  clutch  to  enable 
graceful  handoffs  of  control  by  one  agent  and  rendevous  ot 
control  by  another  agent.  This  is  done  by  simply  having  two 
numan  operators  swap  use  of  the  controls  following 
disengagement  of  the  position  clutch  once  the  forward 
simulation  is  out  well  ahead  of  the  telcrobot.  We  have  compiled 
a  video  report  showing  the  above  experiments,  demonstrations 
and  control  effects  [CON87]. 

5  2.  Plans  for  further  experiments  and  concept  demonstrations 

We  are  continuing  the  above  senes  of  mais,  varying 
additional  system  parameters.  We  are  also  preparing  additions 
types  of  experiments  and  demos.  The  time  brake  and  position 
clutch  will  be  used  tc  determine  thetr  effects  on  specification 
and  manipulation  times,  and  manipulation  path  lengths 
(Skilled  operators  can  use  the  time  brake  to  'erase"  poor  path 
sections,  and  the  position  clutch  to  make  and  "jump  across" 
gaps  dunng  overshoots).  The  time  brake  wall  be  tested  n 
contingencies  (example;  an  obstacle  falls  across  the  plann.  j 
path  behind  the  forward  simulator).  Transitions  involving 
cogmave/mampulanve  task-nesnng  will  be  explored.  We  will 
add  the  command  buffer  and  interpolator  modifications. and 
HCI  controls,  to  implement  and  demonstrate  nme-rauo  control 
during  nme-synchroruzed  forward  simulation. 

We  also  plan  to  attempt  demos  of  simple  forms  of 
role  reversal  by  having  the  tele-automaton  do  the  path 
planning,  and  ierang  the  human  rendezvous  to  telemampulate 
along  selected  pontons  of  the  projected  path.  In  the  role 
reversal  demo  a  route  planner  uses  Al  techniques  to  plan  a  path 
through  a  maze.  The  planner  then  places  the  path  into  the 
forward  simulator  and,  when  necessary,  calls  upon  the  human 
to  take  over  and  manipulate  through  certain  path  segments.  The 
human  then  just  drives  the  manipulator  along  the  displayed  path 
ygment.  This  mimics  a  human  taking  over  the  driving  of  an 
ALV  while  the  ALV  remains  under  machine  cognitive  control. 
This  environment  will  also  enable  demos  ot  human  intervention 
■r,  cognitive  tasks,  for  example  to  assist  in  planning  the  route  if 
■tie  machine  gets  stuck  in  that  high-level  planning  task 

We  found  basic  principles  such  as  Fitt  s  law  very  usefui 
in  thinking  about  forms  of  testbeds  and  hypotheses  for  our 
early  trials.  We  need  to  consider  additional  system  parameters 
and  also  the  dynamics  of  the  human/ machine  combination, 
generate  further  hypotheses  regarding  fa  tors  affecting 
performance,  and  then  design  experiments  to  test  these  ideas. 
Such  work  may  eventually  produce  principles  and  design  rules 
for  icic-autonomous  manipulation  systems. 


<S  FUTURE  RESEARCH  CHALLENGES 

This  initial  tele-automation  work  suggests  •■'pporr  .rcit  , 
for  coordination  of  research  in  several  specialized  :.e:ds  it  a.vo 
mises  issues  concerning  reyarch  equipment  irrrxstru 

Tele-autonomous  technology  presents  new  challenges  ;n 
human  computer  interaction.  We  have  proposed  a  vet  ot 
interface  controls  that  are  conceptually  simple  and  easy  to 
mechanize.  The  controls  are  generic  ones  that  may  be 
applicable  in  many  different  specialized  situations  They  are 
also  cognitively  and  manipulanvely  accessible  to  the  uninitiated 
by  analogy.  But  many  other  new  human  interface  aspects 
haven't  been  pinned  down  at  all.  How  is  the  operator  to 
visualize  where  they  are.  who  has  control  of  what,  and  who 
they  give  control  to  next  as  they  enter  or  leave  some  subtask 
within  a  complex  task  lattice?  What  meases  can  we  provide 
concerning  operator  performance,  and  what  feedback  can  we 
piovide?  And  what  about  the  analysis  and  design  of  cognitive 
and  manipulation  tasks  themselves?  Research  can  perhaps 
provide  better  measures  of  joint  human -machine 
cognmve-manipuianve  performance.  Analyses  similar  to  those 
;n  JCAR83)  may  then  lead  us  to  design  intermixings  of  human 
and  machine  activity  that  yield  substantial  improvements  m 
overall  performance. 

Reyarch  challenges  arise  in  robotics,  such  as  the  eventual 
need  to  perceive,  model  and  forward  simulate  not  only  the 
remote  tele-automaton,  but  also  portions  of  the  remote 
environment  itself.  Forward  simulation  will  work  fine  when 
interacting  with  static  objects,  but  what  about  interactions  with 
moving  objects?  Even  if  we  knew  how  to  specify  interactions 
with  moving  objects,  such  work  would  be  severely  constrained 
by  the  high  computational  complexity  of  present  methods  for 
representing  and  simulating  mechanical  systems.  Further  basic 
work,  such  as  that  of  Hopcroft,  on  efficient  representation  and 
simulation  of  mechanical  systems  is  required  we  are  to 
Kir'ile  problems  of  really  interesting  complexity  iHOP87’ 

More  work  is  needed  on  methods  for  path  error 
vpecificanon  and  associated  methods  for  the  time  optimization 
of  path  following,  such  as  in  [SUH87],  Additional  work  is 
also  needed  on  autonomous  "reflex''  actions  that  the  remote 
robot  can  perform  when  encountering  uncertainties  (particularly 
those  involving  contact)  not  modelled  in  ihe  forward 
simulation.  We  also  need  augmented  AI  programming 
environments  that  interface  in  such  a  way  with  real-time 
programming  environments  as  to  easily  enable  ranid  estimation 
of  orae  available  for  short-term  AI  planning  tasks  (enabling  us 
to  select  among  AI  methods  as  a  function  of  available  ome ). 

We  believe  that  fundamental  work  can  be  done  in  these 
areas  with  modest  robotic  laboratory  equipment.  A I 
techniques  [WLN84)  and  expert  system  technology  [HAY83] 
have  matured  so  that  roboticists  can  now  mechanize 
knowledge-intensive  cognitive  functions  well  beyond  their 
reach  just  a  few  years  ago.  rnd  can  run  these  systems  on 
accessible  workstations.  Thus  mixings  of  manipulation  and 
cognition  technologies  are  now  npe  for  research  exploration. 

However,  some  experiments  will  benefit  from 
multi-dimensional  teleoperators  or  high-tech  automation  or 
autonomous  system  technology.  One  way  to  gain  access  to 
such  expensive  equipment  is  to  treat  remoteness  as  a  feature: 
For  example,  we  are  negotiating  connection  of  our 
tele-autonomous  control  equipment  via  satellite  links  with 


automation  systems  at  several  remote  sites.  As  such  efforts 
provide  useful  testbeds,  others  might  exploit  shared  access  to 
the  same  remote  facilities.  Shared  access  to  capital  equipmcm 
has  obvious  costs  benefits,  but  in  addition  can  stimulate 
standards,  collaborations  and  healthy  direct  competitions 
among  researchers.  Shared  access  to  silicon  foundries  greatly 
increased  the  productivity  of  the  VLSI  research  community 
ICON81J;  a  remote  tele-automation  facility  could  play  an 
analogous  role  in  tele-autonomous  systems  research. 


7  SUMMARY 


We  have  introduced  basic  functional  concepts  for 
tele  autonomous  technology  and  an  architectural  framework  for 
implementing  the  technology,  using  controls  over  rime  and 
position  synchrony  that  enable  simple  structuring  of  control 
transitions.  We  have  proposed  hypotheses  concerning 
capabilities  of  the  technology,  described  our  environment  for 
investigating  these  phenomena,  and  discussed  the  results  of 
early  tests  of  some  of  the  hypotheses.  The  results  indicate  how 
teleroboacs  can  be  extended  to  include  projection  of  cognitive 
activity  and  autonomous  systems  extended  to  accomodate 
smooth  transitions  of  cognitive  or  manipulative  responsibility 
between  machine  and  human  operator.  Through  such 
extensions,  the  two  technologies  merge  into  one  of 
"tele-autonomous  systems".  Finally,  we  have  also  sketched 
some  further  lines  of  research  suggested  by  this  initial  work. 

We  believe  that  tele-autonomous  systems  research  can 
yield  methods  and  systems  for  improved  projection  of 
intelligent,  manipulative  action  at  a  distance  in  rime  and  space. 
This  interdiscipline  presents  interesting  new  research 
opportunities  to  teams  having  expertise  in  robotics  and 
automation,  artificial  intelligence,  and  the  psychology  oi 
human-computer  interaction.  We  envision  many  possible 
applications  for  the  resulting  technology,  not  only  in  space  and 
defense  systems,  but  also  in  design  and  production  systems, 
and  eventually  in  personal  and  recreational  environments. 
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A  BS  TRACT 

A  hierarchical  architectu: a  is 
described  which  supports  space  station 
telerobots  in  a  variety  of  nodes.  The 
system  is  divided  into  three  hierarchies: 
task  decomposition,  world  model,  and 
sensory  processing.  Goals  at  each  level 
of  the  task  decomposition  hierarchy  are 
divided  both  spatially  and  temporally  into 
simpler  commands  for  the  next  lower  level. 
This  decomposition  is  repeated  until,  at 
t.ne  lowest  level,  the  drive  signals  to  the 
robot  actuators  are  generated.  To 
accomplish  its  goals,  task  decomposition 
modules  must  often  use  information  stored 
m  the  world  model.  The  purpose  of  the 
sensory  system  is  to  update  the  world 
model  as  rapidly  as  possible  to  keep  the 
model  in  registration  with  the  physical 
world.  This  paper  describes  the 
architecture  of  the  entire  control  system 
hierarchy  ar.d  how  it  can  be  applied  to 
space  telerobot  applications. 


1.  INTRODUCTION 

One  of  the  ma; or  directions  on  which 
the  robot  research  community  has 
concentrated  its  efforts  is  concerned  with 
planning  and  controlling  motion.  Given  a 
specific  task,  a  motion  plan  must  be 
calculated  which  meets  the  task 
requirements.  Then,  the  plan  must  be 
executed;  there  must  be  sufficient  control 
for  the  robot  to  adequately  effect  the 

desired  motion. 

Trajectories  are  often  planned  as 
straight  lines  in  Cartesian  space  [1]. 
Whitney  '.2.3]  developed  the  resolved 
motion  rate  control  method  for  Cartesian 
straight  line  motions.  Paul  [4,5,6] 
used  homogeneous  coordinate 
transformations  to  describe  a  trajectory 
as  a  function  of  time,  and  Taylor  [7]  used 
coordinated  joint  control  over  small 
segments  to  keep  the  trajectory  within  a 
specified  deviation  of  the  desired 
straight  line  trajectory. 

While  the  research  described  above 
employs  a  "kinematic"  approach  to  robot 
control,  another  direction  of  research 
takes  the  manipulator  "dynamics"  into 
account  in  the  description  of  robot 
motion.  The  dynamic  equations  of  motion 
are  described  either  by  the  Lagrangian 
formulation  [8]  or  by  the  Newton-Euler 
equations  [9].  Algorithms  and  computer 
architectures  have  been  suggested  which 
promise  real-time  dynamic  robot  control 
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Another  aspect  of  motion  control  is 
concerned  with  the  variables  being 
controlled.  The  research  described  to 
this  point  was  concerned  primarily  with 
position  control.  The  robot  moved  from  an 
initial  position  to  a  goal  position. 
While  this  is  perhaps  the  most  common 
mode,  there  are  many  applications  for 
robots  which  suggest  that  other  variables 
should  be  controlled.  For  example,  force 
control  would  be  desired  for  assembly 
operations.  Raibert  and  Craig  [12] 
suggest  a  method  for  hybrid  position/force 
control  of  manipulators. 

These  examples  point  to  the  more 
general  problem  of  sensory  processing. 
For  a  great  deal  of  robot  motion  research, 
sensory  processing  has  been  limited  to 
joint  positions,  velocities,  and 

accelerations.  However,  other  sensors  are 
often  required  to  accomplish  tasks.  The 
control  community  has  concentrated  on  the 
control  aspects  of  the  robot  and  as 
result,  little  emphasis  has  been  placed  on 
sophisticated  sensory  processing. 

Machine  vision,  an  offshoot  of  image 
processing  research,  has  recently  been 
associated  with  advanced  robot 

applications.  One  of  the  most  interesting 
directions  in  this  research  area  is 
concerned  with  sensor  controlled  robots. 
Operating  with  the  constraints  imposed  by 
real-time  robot  control,  early  methods 
used  structured  light  and  binary  images 
[13,14,15,16].  These  approaches,  though 
developed  at  different  institutions, 
shared  many  concepts.  One  of  the 
important  subsequent  research  efforts  went 
toward  the  development  of  model-based 
image  processing.  Bolles  and  Cain  [17] 
used  models  of  objects  to  guide  the 
algorithms  in  a  hypothesis/verification 
scheme  known  as  the  local  feature  focus 
method.  The  concept  has  recently  been 
extended  from  two  dimensional  (i.e.  nearly 
flat)  objects  to  three  dimensional  objects 

[18] .  Although  the  approaches  described 
here  have  led  to  a  better  understanding  of 
real-time  vision  processing,  the  systems 
lacked  a  sophisticated  interconnection 
with  the  robot  control  system. 

The  Automated  Manufacturing  Research 
Facility  (AKRF) ,  developed  at  the  National 
Bureau  of  Standards,  is  a  hierarchically 
organized  small-batch  metal  machining  shop 

[19] .  It  separates  sensory  processing  and 
robot  control  by  a  sophisticated  world 
model.  The  world  model  has  three 
complementary  data  representations.  Lumla 

[20]  describes  the  CAD-like  section  of  the 

model.  Shneier,  Kent,  and  Mansbach  [21] 
describe  the  octree  and  table 


2.2. 


representations  supported  £'/  tne  aoae*. 
The  model  generates  hypotheses  for  the 
features  which  are  either  verified  or 
refuted  by  empirical  evidence.  The 

2 e n s o ry  system's  tasK  is  to  update  the 
appropriate  parts  of  the  world  model  with 
new  or  revised  data  as  rapidly  as 
possible.  The  control  system  accesses  the 
world  model  as  desired  to  obtain.  the 
current  best  guess  concerning  any  aspect 
of  the  world.  Shn.eier,  lumia,  and  Kent 
22'  describe  the  sensory  system  and  its 
operation  in  greater  detail.  The  AMRF  was 
the  first  deliberate  attempt  to  tie 

together  sensory  processing,  world 
modeling,  and  robot  control  in  a  generic 
fashion.  The  system  developed  for  the 
AMRF  is  applicable  to  more  than 

manufacturing.  This  paper  describes  its 
use  in  soace  telerobotics. 


A  FUNCTIONAL  SYSTEM  ARCHITECTURE 


The  fundamental  paradigm  is  shown  in 
Figure  1.  The  control  system  architecture 
is  a  three  legged  hierarchy  of  computing 
modules,  serviced  by  a  communications 
system  and  a  common  memory.  The  task 
decomposition  modules  perform  real-time 
planning  and  task  monitoring  functions, 
and  decompose  task  goals  both  spatially 
and  temporally.  The  sensory  processing 
modules  filter,  correlate,  detect,  and 
integrate  sensory  information  over  both 
space  and  time  in  order  to  recognize  and 
measure  patterns,  features,  objects, 
events,  and  relationships  m  the  external 
world.  The  world  modeling  modules  answer 
queries,  make  predictions,  and  compute 
evaluation  functions  on  the  state  space 
defined  by  the  information  stored  in 
common  memory.  Common  memory  is  a  global 
database  which  contains  the  system's  best 
estimate  of  the  state  of  the  external 
world.  The  world  modeling  modules  keep 
the  common  memory  database  current  and 
consistent. 


Worl  d  Model  mg  -  M 
(Remember,  Estimate, 
Evaluate; 


The  second  leg  of  the  hierarchy 
consists  of  world  modeling  M  modules  which 
model  (i.e.  remember,  estimate,  predict; 
and  evaluate  the  state  of  the  world.  The 
"world  model"  is  the  system's  best 
estimate  and  evaluation  of  the  history, 
current  state,  and  possible  future  states 
of  the  world,  including  the  states  of  the 
system  being  controlled.  The  "world 
model"  includes  both  the  M  modules  and  a 
knowledge  base  stored  m  a  common  memory 
database  where  state  variables,  maps, 
lists  of  objects  and  events,  and 
attributes  of  objects  and  events  are 
maintained.  3y  this  definition,  the  world 
model  corresponds  to  what  is  widely  known 
throughout  the  artificial  intelligence 
community  as  a  "blackboard"  (23;  .  The 
world  model  performs  the  following 
functions : 


1.  Maintain  the  common  memory 
knowledge  base  by  accepting 
information  from,  tn.e  sensory 
system . 


2.  Provide  predictions  si  expected 

sensory  input  to  the 

corresponding  G  modules,  based 

on  the  state  of  the  task  and 
estimates  of  the  external  world. 

3.  Answer  "What  is?"  questions 

asked  by  the  executors  in  the 
corresponding  level  H  modules. 
The  task  executor  can  request 

the  values  of  any  system 

variable . 


4.  Answer  "What  if?"  questions 
asked  by  the  planners  in  the 
corresponding  level  H  modules. 
The  M  modules  predict  the 
results  of  hypothesized  actions. 


2.1.  Task  Decomposition  -  H  modules 

(Plan,  Execute) 

The  first  leg  of  the  hierarchy 
consists  of  task  decomposition  K  modules 
which  plan  ar.d  execute  the  decomposition 
of  high  level  goals  into  low  level 
actions.  Task  decomposition  involves  both 
a  temporal  decomposition  (into  sequential 
actions  along  the  time  line)  and  a  spatial 
decomposition  (into  concurrent  actions  by 
different  subsystems).  Each  H  module  at 
each  level  consists  of  a  job  assignment 
manager  JA,  a  set  of  planners  PL(i),  and  a 
set  ci  executors  EX(i).  These  decompose 
the  input  task  into  both  spatially  and 
temporally  distinct  subtasks  as  shown  in 
Figure  2.  This  will  be  described  in 
greater  detail  in  section  4. 


2.3.  Sensory  Processing  -  G  modules 

(Filter,  Integrate ,  Detect , 
Measure) 

The  third  leg  of  the  hierarchy 
consists  of  sensory  processing  G  modules. 
These  recognize  patterns,  detect  events, 
and  filter  and  integrate  sensory 
information  over  space  and  time.  The  G 
modules  at  each  level  compare  world  model 
predictions  with  sensory  observations  and 
compute  correlation  and  difference 
functions.  These  are  integrated  over  time 
and  space  so  as  to  fuse  sensory 
information  from  multiple  sources  over 
extended  time  intervals.  Newly  detected 
or  recognized  events,  objects,  and 
relationships  are  entered  by  the  M  modules 
into  the  world  model  common  memory 
database.  and  objects  or  relationships 
perceived  to  no  longer  exist  are  removed. 
The  G  modules  also  contain  functions  which 
can  compute  confidence  factors  and 
probabilities  of  recognized  events,  and 
statistical  estimates  of  stochastic  state 
variable  values. 


; .  4  .  Operator  Interfaces 

:ControI,  Observe,  Define  Goals, 
Indicate  Objects) 

The  control  architecture  defined 
r.ere  has  an  operator  interface  at  each 
level  in  the  hierarchy.  The  operator 
interface  provides  a  means  by  which  human 
iterators ,  either  in  the  space  station  or 
on  the  ground ,  can  observe  and  supervise 
the  telerobot.  Each  level  of  the  task 
decomposition  hierarchy  provides  an 
interface  where  the  human  operator  can 
assume  control.  The  task  commands  into 
any  level  can  be  derived  either  from  the 
higher  level  H  module,  or  from  the 
operator  interface.  Using  a  variety  of 
input  devices  such  as  a  joystick,  mouse, 
trackball,  light  pen,  keyboard,  voice 
input,  etc.,  a  human  operator  can  enter 
the  control  hierarchy  at  any  level,  at  any 
time  of  his  choosing,  to  monitor  a 
orocess,  to  insert  information,  to 
interrupt  automatic  operation  and  take 
control  of  the  task  being  performed,  or  to 
apply  human  intelligence  to  sensory 
crocessing  or  world  modeling  functions. 

The  sharing  of  command  input  between 
human  and  autonomous  control  need  not  be 
ail  or  none.  It  is  possible  in  many  cases 
for  the  human  and  the  automatic 
controllers  to  simultaneously  share 
control  of  a  telerobot  system.  For 
example  a  human  might  control  the 
orientation  of  a  camera  while  the  robot 
automatically  translates  the  same  camera 
through  space. 

2.4.1  Operator  Control  interface  levels 

The  operator  can  enter  the  hierarchy 
at  any  level.  The  operator  control 
interface  interprets  teleoperation  in  the 
fullest  sense:  a  teleoperator  is  any 
device  which  is  controlled  by  a  human  from 
a  remote  location.  While  the  master-ilave 
paradigm  is  certainly  a  type  of 
teieoperation,  it  does  not  constitute  the 
only  form  of  man-machine  interaction.  At 
different  levels  of  the  hierarchy,  the 
interface  device  for  the  human  may  change 
but  the  fundamental  concept  of 
taleoperation  is  still  preserved.  Table  1 
illustrates  the  interaction  an  operator 
may  have  at  each  level. 

The  operator  control  interface  thus 
provides  mechanisms  for  entering  new 
instructions  or  programs  into  the  various 
control  modules.  This  can  be  used  on-line 
for  real-time  supervisory  control,  or  in  a 
background  mode  for  altering  autonomous 
teierobot  plans  before  autonomous 
execution  readies  that  part  of  the  plan. 

2.4.2  Operator  monitoring  interfaces 

The  operator  interfaces  allow  the 
human  the  option  of  simply  monitoring  any 
level.  Windows  into  the  common  memory 
knowledge  base  permit  viewing  of  maps  of 
service  bay  layout,  geometric  descriptions 
and  mechanical  and  electrical 
configurations  of  satellites,  lists  of 


recognized  objects  and  events,  object 
parameters,  and  state  variables  such  as 
positions,  velocities,  forces,  confidence 
levels,  tolerances,  traces  of  past 
history,  plans  for  future  actions,  and 
current  priorities  and  utility  function 
values.  These  may  be  displayed  in 
graphical  form,  for  example  using  dials  or 
bar  graphs  for  scalar  variables,  shaded 
graphics  for  object  geometry,  and  •. 
variety  of  map  displays  for  spatial 
occupancy . 

2.4.3  Sensory  processing/world  modeling 
interfaces 

The  operator  interface  may  also 
permit  interaction  with  the  sensory 
processing  and/or  world  modeling  modules. 
For  example,  an  operator  using  a  video 
monitor  with  a  graphics  overlay  and  a 
light  pen  or  joystick  might  provide  human 
interpretative  assistance  to  the 
vision/wcrld  modeling  system.  The 
operator  might  interactively  assist  the 
model  matching  algorithms  by  indicating 
with  a  light  pen  which  features  in  the 
image  (e.g.  edges,  corners)  correspond  t; 
those  in  a  stored  model.  Alternatively, 
an  operator  could  use  a  joystick  to  line 
up  a  wireframe  model  with  a  TV  image, 
either  in  2-D  or  3-D.  The  operator  might 
either  move  the  wireframe  model  so  as  to 
line  up  with  the  image,  or  move  the  camera 
position  so  as  to  line  up  the  image  with 
the  model.  Once  the  alignment  was  nearly 
correct,  the  operator  could  allow 
automatic  matching  algorithms  to  complete 
the  match,  and  track  future  movements  of 
the  image. 

2.5.  Common  Memory 

2.5.1.  Communications 

One  of  the  primary  functions  of 
common  memory  is  to  facilitate 
communications  between  modules. 
Communications  within  the  control 
hierarchy  is  supported  by  a  common  memory 
in  which  state  variables  are  globally 
defined. 

Each  module  in  the  sensory 
processing,  world  modeling,  and  task 
decomposition  hierarchies  reads  inputs 
from,  and  writes  outputs  to,  the  common 
memory.  Thus  each  module  needs  only  to 
know  where  in  common  memory  its  input 
variables  are  stored,  and  where  in  common 
memory  it  should  write  its  output 
variables.  The  data  structures  in  the 
common  memory  then  define  the  interfaces 
between  the  G,  M,  and  H  modules. 

The  operator  interfaces  also  interact 
with  tne  system  through  common  memory. 
The  operator  displays  simply  read  the 
variables  they  need  from  the  locations  in 
common  memory.  If  the  operator  wishes  to 
take  control  of  the  system,  he  simply 
writes  command  variables  '  to  the 
appropriate  locations  in  common  memory. 
The  control  modules  that  read  from  those 


locations  need  not  know  whether  their 
input  commands  derived  from  a  human 
operator,  or  from  the  next  higher  level  m 
the  autonomous  control  hierarchy. 

2.5.2  State  Variables 

The  state  variables  m  common  memory 
are  the  system's  best  estimate  of  the 
state  of  the  world,  including  both  the 
external  environment  and  the  internal 
state  of  the  H,  M,  and  G  modules.  Data  in 
common  memory  are  available  to  ail  modules 
at  ail  levels  of  the  control  system. 

The  knowledge  base  in  the  common 
memory  consists  of  three  elements:  maps 

which  describe  the  spatial  occupancy  of 
the  world,  object-attribute  linked  lists, 
and  state  variables. 

3.  LEVELS  IN  THE  CONTROL  HIERARCHY 

The  control  system  architecture 
described  here  for  the  Flight  Telerobot 
System  is  a  3ix  level  hierarchy  as  shown 
in  Figure  3.  At  each  level  in  this 
hierarchy  a  fundamental  transformation  is 
performed  on  the  task. 

Level  1  transforms  coordinates  from  a 
convenient  coordinate  frame  into 
joint  coordinates.  This  level 
also  servos  joint  positions, 
velocities,  and  forces. 

Level  2  computes  inertial  dynamics,  and 
generates  smooth  trajectories  in 
a  convenient  coordinate  frame. 

Level  3  decomposes  elementary  move 
commands  (E-moves)  into  strings 
of  intermediate  poses.  E-moves 
are  typically  defined  in  terms 
of  motion  of  the  subsystem  being 
controlled  (i.e.,  transporter, 
manipulator,  camera  platform, 
etc.)  through  a  space  defined  by 

E-move  commands  may  consist  of 
symbolic  names  of  elementary 
movements,  or  may  be  expressed 
as  keyframe  descriptions  of 
desired  relationships  to  be 
achieved  between  system  state 
variables.  E-moves  are 

decomposed  into  strings  of 
intermediate  poses  which  define 
motion  pathways  that  have  taen 
checked  for  clearance  with 
potential  obstacles,  and  which 
avoid  kinematic  singularities. 

Level  4  decomposes  object  task  commands 
specified  in  terms  of  actions 
performed  on  objects  into 

sequences  of  E-moves  defined  in 
terms  of  manipulator  motions. 
Object  tasks  typically  define 

actions  to  be  performed  by  a 
single  sultiarmed  telerobot  sys¬ 
tem  on  one  object  at  a  time. 
Tasks  defined  in  terse  of 
actions  on  objects  ars 


decomposed  into  sequences  of  E- 
moves  defined  in  terms  of 
manipulator  Dr  vehicle  subsystem 
motions.  This  decorcpos  n  ic.n 
checks  to  assure  that  there 
exist  motion  freeways  clear  of 
obstacles  between  keyframe 
poses,  and  schedules  coordinated 
activity  of  telerobot 

subsystems,  such  as  the 
transporter,  dual  arm 

manipulators,  mult i fingered 

grippers,  and  camera  arms. 

Level  5  decomposes  actions  to  be 
performed  on  batches  of  parts 
into  tasks  performed  on 
individual  objects.  It 

schedules  the  actions  of  one  or 
more  telerobot  systems  to 
coordinate  with  other  machines 
and  systems  operating  in  the 

immediate  vicinity.  For 

example,  Level  5  decomposes 
service  bay  action  schedules 
into  sequences  of  object  task 
commands  to  various  telerobot 
servicers,  astronauts,  and 
automatic  berthing  mechanisms. 
Service  bay  actions  a.e 
typically  specified  in  terms  of 
servicing  operations  to  be 
performed  by  all  the  systems 

(mechanical  and  human)  in  a 
service  bay  on  a  whole 

satellite.  This  decomposition 
typically  assigns  servicing 
tasks  to  various  telerobot 
systems,  and  schedules  servicing 
tasks  so  as  to  maximize  the 
effectiveness  of  the  service  bay 
resources . 

Level  6  decomposes  the  satellite 

servicing  mission  plan  into 
service  bay  action  commands. 
Mission  plans  are  typically 
specified  in  terms  of  satellite 
servicing  priorities, 

requirements,  constraints,  and 
mission  time  line.  The  level  6 
decomposition  typically  assigns 
satellites  to  service  bays,  sets 
priorities  for  service  bay 
activities,  generates 

requirements  for  spare  parts  and 
tool  kits,  and  schedules  the 
activities  of  the  service  bays 
so  as  to  maximize  che 
effectiveness  of  the  satellite 
servicinq  mission.  To  c  large 
extent  the  level  6  mission  plans 
will  be  generated  off  line  on 
the  ground,  either  by  human 
mission  planners,  or  by 
automatic  or  semiautomatic 
mission  planning  methods. 


DETAILED  STRE'CTVRf  v 

MODULES 


i_‘"e  "  module  at  each  level  consists 

;■  -"tee  parts  as  shown  in  Figure  .;  ;  a 
2  ass  *vf'~er't  "ar.ager  JA ,  one  or  more 
~-.arr.ers  PL  .  s  ;  ,  and  one  or  mere  executors 

-A  SI. 

.r.e  ;  rb  assignment  m.anaaer  JA  is 
lespor.sin.e  tor  partitioning  the  tas- 

lomiiand  PC  into  s  spatially  or  logically 
distinct  jots  to  oe  performed  by  s 
physically  distinct  planr.er/executor 

•teen;  ..isms .  At  the  upper  levels  the  job 

assignment  module  may  also  assign 

physical  resources  against  task  elements. 
The  output  of  the  gob  assignment  manager 
is  a  set  of  job  commands  JC(s),  s=l .  2, 

or  logically,  distinct  gobs. 


where  n  =  the  number  of  state  clock 
periods  required  to  compute  the 
function  EX(s) .  n  typically  equals  1.  The 
feedback  F3(s,t)  also  carries  timing  and 
subgoal  event  information  for  coordination 
of  output  between  executors  at  the  same 
level .  When  the  executor  detects  a 
subgoal  event,  it  selects  the  next  planned 
subtask  from  the  queue. 

Executor  output  5TX's,t)  also 
contains  requests  for  information  from  the 
world  model  M  module,  and  status  reperts 
to  the  next  higher  (i-1)  level  in  the  H 
module  hierarchy.  The  feedback  FB(s,t) 
contains  status  reports  from  the  H  module 
at  the  i-1  th  level  indicating  progress  on 
its  current  task. 

5.  CONCLUSION 


For  each  of  these  job  commands  JC(s), 
th.e.e  exists  a  planner  PL(s)  and  a 
executor  EX(s).  Each  planner  PL(s)  is 
responsible  for  decomposing  its  job 
command  ^C(s)  into  a  temporal  sequence  of 
planned  subtasks  ?ST(s,tt).  Planning 
typically  requires  evaluation  of 
alternative  hypothetical  sequences  of 
planned  subtasks.  The  planner 
hypothesizes  some  action  or  series  of 
actions,  the  world  model  predicts  the 
results  of  the  action(s)  and  computes  some 
evaluation  function  EFfs.tt)  on  the 
predicted  resulting  state  of  the  world. 
The  hypothetical  sequence  of  actions 
producing  the  best  t  1  nation  function 
EF's,tt)max  is  then  selected  as  tne  pl..,n 
PSTi's.tt)  to  be  executed  by  the  executor 
EX  ( s ;  . 

PST's.tt,  =  PL's)  ;JC(s; ,EF(s,tt)max: 

where  tt  is  the  time  sequence  index  for 
steps  in  the  p.an.  tt  may  also  be  defined 
as  a  running  temporal  index  in  planning 
space,  tt  =  1,  2,  . . . ,  th  wnere  th  is 
the  value  of  the  tt  index  at  the  planning 
horizon.  The  planning  horizon  is  defined 
as  tne  period  into  the  future  over  which  a 
plan  *s  prepared.  Each  level  of  the 
hierarchy  has  a  planning  horizon  of  one  or 
two  expected  input  task  time  durations. 

Each  executor  EXrs)  is  responsible 
for  successfully  executing  the  plan 
?ST  '  s , it )  prepared  by  its  respective 
planner  PL's; .  If  all  the  subtasks  in  the 
plan  PST  (  s , tt )  are  successfully  executed, 
then  the  goal  of  the  original  task  will  be 

achieved.  The  executor  operates  by 
selecting  a  subtask  from  the  current  queue 
of  planned  subtasks  and  outputting  a 
subcommand  STX(s,t)  to  the  appropriate 
subordinate  H  module  ut  time  t.  The  EX(s) 
module  monitors  its  feedback  FB(s,t)  input 
in  order  to  3ervo  its  output  STX(s,t)  to 
the  desired  subtask  activity. 

31’X's,  t-n)  -  EX'S)  [  PST  (  s  ,  t )  ,  FB  (  a  ,  t )  j 


This  paper  l.as  described  a  hierarchical lv 
organized  control  systam  and  has  shown  how 
this  generic  system  can  be  applied  to 
telerobotic  applications  in  space  by 
considering  the  requi -  aments  of  a  flight 
telerobotic  servicer  lor  the  space 
station . 
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TABLE  1 — OPERATOR  INTERACTION  AT  EACH 
LEVEL 


LEVEL  TYPE  OF  INTERACTION 

At  the  servo  replica  master,  individual 


joint  position,  rate,  or 
force  controllers. 

above  level  1  ]cy  stick  co  perform 
resolved  motion  force/rate 
control 

above  level  2  indicate  safe  motion 

pathways.  Icbot  ecu 
dynamically  e  f f i 

.cenents 


above  level  3  graphically  or  symbolically 
define  key  poses.  menus  to 
choose  elemental  moves. 

above  level  4  specify  tasks  to 
performed  on  objects. 

above  level  3  reassign  telerobots  to 
different  service  bavs. 
insert,  modify,  and  aonitoi 
plans  describing  servicing 
task  sequences. 

above  level  6  reconfigure  servicing 

mission  priorities. 


a  a 


Task  Decomposition 


Decomposition 

FIGURE  2:  The  job  assignment  Jl'  perform  a  snatial  c’ecorpos  it  ion  of  the  task.  The 
planners  PI  (jt  and  executors  FX  (j)  perforn  a  temporal  decorpos it  ion 


% 


ACTUATORS - 


FIGURE  3:  A  six  level  Kierarchial  Control  Evster.  Propoeed  for 
Multiple  Autonomous  Vehicles 
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Turing  ire  iast  two  years,  ir.e  V.  3.  Ar~y  -as 
.-ple-erced  a  new  cnogram.  io  develop  ire  tec -no  1 - 
gy  case  required  for  robotic  systems  vnip.n  :ar 
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'  e  'ess  i  •  at  tog  maximum  productivity  f-cm  ava.iacie 
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te  '  t85,  Che  Army  Materiel  Demand 
asked  tr.e  "J.  3.  Amry  Human 
donatory  HEL;,  AMC's  lead  agency 
"ted  robotics,  to  develop  a  thrust 
fits  which  would  achieve  "critical 
v  gey  programs.  The  .-“suiting  ore— 
h  tr.e  Army's  Arroyo  Center  Hand 
deral  Contract  Researon  Center! 
tical  support,  is  a  broad  based 
eg  a  diverse  array  of  research  and 
civicies  from  tne  Army'  •,  Materiel 
and  Training  and  Doctrine  Command 
Defense  Advanced  Research  Projects 
,  National  Bureau  of  Standards, 
suen  as  Cak  Ridge  and  Sandia,  and  a 
ersities.  The  pace  and  scope  of  the 
t  requires  utilization  of  suoport 
c  identified  above  and  the  contractor 
rrent  industry  participation  in  the 
es  some  of  the  largest  corporations 
as  well  as  small  innovative 


ur.vrinh-  1  >R7  by  -.  y.  Shoemaker. 
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>f  Aeronautics  and  As  t  ronaur  : -s  ,  Tns~ 
with  uermi ss ion . 
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*i*rv<?r. *1 1 2r.  _r:  ‘r,e  3vs*5?n  1  3  > 0 r.  «  1 . . 
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ar.  *  t r. r  1  *. 0 <7 v  wr'i'th  1  r. c  1. H. ^ s  -r* ^ ^ . r  ■  ~  ■ 

: r.  r i f- ^ 1 1  t. £*  i~r. 

r':  ‘i3k  ;s  rr‘a~e'~*. 


•  r c r : 2 . ^ r  '.re  Po&ctics  Tr,r'js‘ .  >.  ‘re 

le.'l  ee  wide  ran*®  of  appiicatior  :  r  ~r‘r''-,r'  *  1 

tre  ’ ecmclogy .  In  fact,  this  is  a  — al . 

-epnesen"  at  i/e  subset  of  the  application  •  — .•  .  ‘  • 
interests  :n  robotics.  .•  Next,  the  ap 
nave  seen  parsed  into  teleoperator  an 
systems.  Although  •eieoperaced  systems  it  no ‘ 
tr.e  strict  definition  of  robots  due  tc  tnem  i:w 
of  autonomous  capabilities,  they  play  an  important 
part  m  tne  program  as  .near-term,  ;umr '  r.g-o‘‘f  pc  me 
for  truly  robotic  systems.  Next,  the  gev  technical 
areas  associated  with  the  use  of  robotics  m  ‘re 
field  environment  are  listed.  The  three  taper 
components  of  tne  program  are  listed  under  the 
column  headed  thrusts.  A  brief  description  tf  eacr 
program  follows: 
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Fig.  2  Teleoperated  Mobile  Anti-Armor  Project  (TMAP) 


Material  Handling  Robotics 

This  program  is  focussed  on  robotic 
nan  isolator  systems.  One  of  the  two  program  sub- 
elements,  tre  Field  Material  Handling  Robot  ( FMR  ) , 
will  operate  in  a  logistics  workcell  autonomously 
moving  a  wide  variety  of  palletized  load3  weighing 
ip  to  ?  tons,  with  a  reach  of  27  feet  and  a  load 
rand  ling  cycle  time  of  20  seconds. 


environment,  and  packaging  of  the  system  within 
C-141  aircraft  weight  and  volume  transportability 
constraints.  Following  two  years  of  preparatory 
work  by  the  Human  Engineering  Laboratory,  the 
National  Bureau  of  Standards  and  Belvoir  Research 
and  Development  Center,  a  contract  was  awarded  to 
Martin  Marietta  Aerospace  Corporation  in  May  of 
1986  for  development  of  a  full-scale  FMF  demon¬ 
strator  system.  Although  not  elements  of  the 
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or^ser-,  two  ttrer  fj-ur0  VHE  rorot:t3 

.r.c  '.  ;de  venizle  systems  which  may  vl  t  imate. y  inter¬ 
face  with  the  PieM  Maternl  H-inJl  in*  ?ot> or. 
first  of  tries*3  is  the  Palletized  l,oadinfl  System 
-?LS!  shown  m  Figure  ^ . 


ft:-.  ;rs  i  rives  t  nr  at :  :-ns  i  n  : 
:ccn  is  exclusive  ordnance 


The  ?LS  is  3  transportation  asset  wnich 
presents  the  opportunity  Tor  automated  configuring 
of  flat  racks  of  mixed  materiel  for  the  field  user. 
Rather  than  a  flat  rack  consisting  entirely  of 
artillery  cro'ectilss,  the  situation  may  require  a 
more  seif  contained  lead  of  projectiles,  propellant 
charges,  and  fates  sent  directly  to  firing  units. 

An  illustrative  plan  view  of  such  a  workcell  Is 
shown  m  figure  t. 


Loading  Z y stem 

refueling  of  itmtaf  vehicles,  dec  mfan.r 
tattle  damage  assessment,  key  elements  : 
system,  e.g.,  manipulator,  wrist  and  ere 
r.ar.ds )  are  being  configured  to  ensure  - 
technology  transfer  into  appropriate 
A  xey  technical  achievement  related  ‘o  ; 
teen  'he  lemcnst ration  of  an  active  “an 
deflection  compensation  system  permit* in 
tartly  higher  manipulator  ray  it  ad  •  -  ■ 


snbtti 


SEMI-AUTO 

LIFT-TRUCKS 


SWITCH -CATES 


Fig.  5  PL5  Automated  workcell 


An  alternative  Field  Material  Handling  Rohr* 
use  :3  m  conjunction  with  a  highly  automated 
ammunition  resupply  vehicle  for  the  upload,  trans¬ 
port  and  offload  of  ammunition  for  the  artillery 
us^r.  The  illustration  captioned  artilLery  ammu¬ 
nition  flow  depicts  a  conceptual  application  of 
robotics  to  the  ammunition  supply  system. 

A  second  element  of  the  material  handling 
robotics  program  is  the  Soldier  Robot  Interface 
Project  TRIP).  Tee  Figure  7. 

l*us  program  ls  developing  a  7-foot  reach, 

'TO -lb  robot  manipulator  mounted  on  a  vehicle 
'onnected  to  a  remote  operator  via  fiber  optics. 
The  .TRIP  emphasises  the  otner  end  of  the  manipu¬ 
lator  capability  spectrum  from  the  Field  Material 


than  had  previously  been  the  case.  TRIP  is  an 
example  of  direct  synei^y  utv«-:.c  the  nuclear 
community  and  Army  interests  in  field  applications 
for  robotics.  The  Army  selected  Cak  Ridge  National 
Laboratory  as  the  system  integrator  for  the  TRIP 
project  in  recognition  of  truer  vast  teleoperat lin 
experience  base  in  the  nuclear  community.  The  Army 
Laboratory  Command  is  the  developer  for  Doth  of  tr.e 
material  handling  robot  systems. 

The  Army  will  significantly  benefit  from  the 
DARPA  Advanced  Manipulator  Systems  ( AMS  I  program 
which  will  lead  to  improvements  in  the  design  of 
manipulators  for  field  applications.  Specific 
improvements  are  expected  in  force  control,  more 
dextrous  multi-functional  grippers  and  overall 
improvements  in  system  performance,  packaging  and 
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Manipulator  Arm  (?es  c^  issues 


X .reraticaily-retiundant  articulated  manipulators  ctfer 
s  gn- ’leant  advantages  m  tool-handling  dexterity  and 
ut:  ay.  This  paper  discusses  a  7  DCF  manipulate 
system  m  production  for  terrestrial  factory  automation 
aopiicat'cns  and  the  design  changes  proposed  to 
-are  the  system  appropriate  for  Space  Station  use. 
Tn  s  manipulator  system  features  modu'ar  construction, 
seif-contained  ;omt-mour.ted  electric  dr,ves,  high 
repeatability,  and  the  pr-antial  for  extraordinary 
dead-reckormg  accuracy.  Manipulators  of  varying 
sice,  capacity  and  redundancy  may  be  buiit  'rem  a 
family  of  modules  r.avirg  output  torques  'rom  40  m-lbs 
‘o  40.0CO  m-lUs.  The  con'rol  has  a  helrarchicat 
architecture  designed  for  real-time,  sensor-dnven 
operation  and  employs  an  efficient  proprietary 
algorithm  which  performs  the  transformation  of  6  DOF 
cartesian  space  commands  into  7+  DOF  jomtspace. 
This  algorithm  utilizes  one  or  more  criteria  to  resolve 
'me  redundancy,  distributing  motion  gracefully  to  deal 
with  singularities  and  controlling  the  pose  of  t  e 
manipulator  m  the  u  orkspace.  The  control  algorithm 
rjns  effectively  or,  single  board  microprocessor 
hardware.  This  manipulator  system  is  seen  by  the 
authors  to  be  a  close  fit  to  space  telerobotics 
specifications,  with  existing  models  providing  ready 
means  for  terrestrial  testing  and  demonstration 

Introduction 


A  redo!  c  manipulator  >s  a  powered  mec-arnca  -■sage 
operating  under  interactive  computer  treg'am  o:",rC! 
oseo  to  position  onent,  and  apply  forces  ara  tongues  to 
a  too;  mounted  to  the  distal  era  o'  ,t-e  "-age  a 
number  ot  different  designs  for  suen  se'vomec'a n  s~s 
Pave  Peen  developed,  principally  ‘o-  the  automat  cn  of 
industrial  processes  related  to  autcmcoi  e 
manufacturing,  but  also  for  remote  manrbuia:  c'  -as*s 
undersea  and  m  hizard'-s  radiat  on  en  ^ '■events 
Tne  MASA/Spar  Aerospace:M  a,m.  or  Re~c.e 
Man.-pu'ator  System  (RMS),  is  the  on  y  soace  c-a  •  eu 
-ampu.'ator  ex-tant.  RMS  has  clearly  cemor'shatec  :-e 
uti.ity  of  suen  machines  in  the  new  env.ronmen; 

Established  mechanical  arm  designs  differ  ‘rom  o-^ 
another  m  the  number  and  type  of  .amts  mwcorated 
and  m  their  linkage  geometri  u,  m  the  piacemem  a^a 
type  of  joint  actuators,  and  in  control  architecture 
employed.1  Industrial  robot  arms  available  tocay  ~ay 
have  as  few  as  three  driven  joints,  or  as  many  as 
seven.  In  principle,  the  more  joints  in  the  arm  'mkage. 
the  more  general  purpose  and  versatJe  the 
manipulator.  An  arm  must  incorporate  at  ;east  s  x 
degrees  of  freedom  to  provide  ccmolete  control  of 
position  and  orientation  of  the  tool.  It  is  worth  reviewing 
the  evolution  of  current  mechanical  arm  oes'gns  w: 
considering  the  application  of  estaDiished  art  to  soace 
automaton. 


Mechanical  arms  will  be  among  the  key  functional 
e  ements  m  a  variety  of  different  robotic  systems 
employed  m  ‘uture  soace  operations  by  NASA,  the 
USAF  and  mdustry  In  the  !~30s.  robotic  devices  with 
arms  will  assist  m  the  assembly,  inspection,  and 
maintenance  of  the  Space  Station,  and  m  the  servicing 
u;  satellites  .n  geosynchronous  orbit.  Early  in  the  next 
century,  more  sophisticated,  elaborate  and  capable 
versions  of  these  devices  will  play  important  roles  m 
the  exploration  and  commercial  development  of  the 
Moon  and  solar  system.  Robots  will  serve  as  new  tools 
‘or  the  Astronauts,  augmenting  their  capabilities  and 
making  their  jobs  easier  and  safer.  The  next  phases  ot 
space  exploration  and  development  will  introduce 
human  beings  m  large  numbers  to  the  space 
environment,  leading  to  the  establishment  of 
pormanent  space  communities.  It  is  the  authors' 
conviction  that  the  appropriate  application  of  robotics 
tecnnology  will  accelerate  that  process. 
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Arm  Geometries.  Ac'-ator  Aira r^emem  and  Mot  O'' 
Controllers  forGererai  Purpose  Mar-pUato's 

Six  or  more  degrees  of  freedom  can  be  disposed  n  an 
arm  m  several  ways.  Essentially  the  differences  re  ate 
to  the  geometry  of  the  mechanism  used  to  translate  a 
three  degree  of  freedom  wrist,  or  tool  onenter,  and  the 
type  of  motion  control  system  thereby  required  fc 
produce  straight  line  moves  and  other  controlled  paths 
at  the  toolpoint. 

One  basic  form  of  manipulator  employs  a  set  of  three 
slides  connected  at  prismatic  or  sliding  joints  disposed 
m  a  nominally  orthogonal  arrangement  to  position  the 
wrist  Such  a  device  is  typified  by  the  IBM  RS  1™ 
industrial  robot.  This  "cartesian''  geometry  has  a 
number  of  distinct  advantages  over  other  types,  moot 

BM  is  a  registered  trademark  ot  International  Business  Machines  carp 
Spar  Aerospace  is  a  registered  trademark  ot  Spar  Aerospace  Limited 


•eq-'ea  '3  generate  ..setui  metiers  at  t-e  tocpc-i't 

L_  -ear  a- c  c  m^ar  -terpciattcr  are  suf‘  cem. 

a  second  type  c'ov  ces  a  pa  '  of  s  ces  cc-"ec!ed  a*  i 
t'  :  ~at  c  p  rt.  'Ca'ec  or  a  oa_e  'eve  -‘e  or  p  vet 
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-  a  'c  co — ro”  ama-ge-e-t.  toe  w  st  s  pos  t  c-eo 

-  t-e  .‘.o'-ocaoe  py  a  5  ce  cce-ectea  a:  a  5  -g  e 

c'3~atc  or-;  to  a  aa  r  3?  3rt-cgcra..y  mourned 
•e  .c  Me  0  rts  T-  s  ar.-a-gerrent  was  estaDksred  a-d 
s  typoea  oy  fhe  Ur.rrtation  Umrrate  '000™  industr-a' 
'-oc:  Theoretically,  such  a  "polar"  coordinate 

geometry  produces  a  spnerical  worKspace  In  prance. 
~'scrar-.cai  design  constraints  generally  restr.ct  tne 
^seC  work, no  envelope  to  a  rattier  torn  soner'cal  sne  1 
e;s  canam  3  gn  -  cant  ceric  sectors 

Mere  3 con  st  cated  mot  on  control  systems  are  'ecuireo 
»pr  arms  witn  cyLrd'ical  or  polar  geometries  man  ‘or 
t-ose  -avi-g  cartes, an  geometries,  sm.ee  cooro.rate 
t'ans'crmat.ons  must  ae  oerc'mea  to  generate  sma-grt 
-e  ~pves  a:  me  toc.ocmt  in  acc,:  on.  mamo-  atcrs 
.wen  emp.oy  one  or  mp-e  slides  witn  pr,smatiC  joints 
n  meir  -xage.  as  a  class,  exn.oit  certain  significant 
oe'‘orrance  imitations  of  particular  importance  m 
p'ospecfive  space  applications.  The  relatively  large 
s  ze  and  media  o*  the  sides  result  >n  reiative'y  high 
—o*  ve  power  requirements  ‘or  a  manipulator  of  given 
oayioad  and  workspace.  Furthermore,  the  posrncnmg 
s  ces  erten  interfere  with  oDiects  .n  the  working  area, 
wren  emenced.  and  with  cojects  behind  the  maemr.e. 
wren  'etracted. 

A~org  'me  types  of  manipulators  which  incorporate 
3  ces  tne  cartesian  systems  tend  to  be  the  'east 
soavaiy  e'‘c;ent.  since  tne  workspace  is  often 
completely  surrounded  by  the  framework  of  slides  and 
supporting  structure  To  minimize  spatial  inefficiency 
ana  interference  problems,  a  few  polar  type  arms  have 
oeen  designed  in  which  the  slide  collapses  on  itself  as 
t  s  'etracted.  In  the  U  S  Robot  Maker™,  for  example, 
•re  side  s  composed  or  a  set  of  shorter  slides  which 
•e  escape.  In  me  NASA/Martin  Marietta™  Viking 
Lander  arm,  a  mm-wail  steel  tube  that  forms  the  slide 
,vn en  extended  3  made  to  collapse  flat  m  cross-section 
py  'Oilers  as  t  s  'etracted.  where  it  can  be  coiled 
compact  y  on  a  drum.  Mechanical  implementations  of 
•-ese  ces  grs  tend  to  have  comparatively  poor  static 
and  dynamic  performance  characteristics,  however, 
d^e  either  to  me  number  of  additional  prismatic  joints 
neerporated  or  to  the  very  thm-wall  cross-section 
M-ized. 

To  mpreve  dynamic  performance  and  solve  workspace 
nterferenco  propiems,  a  linkage  geometry  wnicn 
permits  considerably  more  efficient  mechanical 
designs  has  Deen  devised  and  widely  adopted.  In  this 
geometry,  a  series  of  relatively  rigid  link  segments 
connected  at  revolute  joints  is  used  to  position  the 
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0  -red  arm'  cr  "ad'cuiatea"  manpu  ato'  T-ey  va 
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:-e-e  s  o-e  or  at  most  two  aspect  arm  conf-g^rat  c  s 
■pr  any  g  ven  tool  'ocaf.on  a-d  a'*  'uce  T  -e 


advantages  to  oe  derived  ‘rp.m  "'co'po'aV'g  e<"a 
c  -'s  -  an  articulated  arm.  "kinematic  'edundar.cy' 
w  oe  discussed  n  suosequent  paragraphs 

Tnecreticaily,  a  jemted  arm  design  of  s,x  or  —  me 
deg'ees  of  freedom  produces  a  'cugMy  sphercai 
werkmg  envelope.  The  pnnc'pai  advantages  0'  a 
,o.n;ed  arm  manipulate',  of  oamcu  ar  t 
ge-eral-purpose  arms  ‘cr  space  cpe'at  c-s.  -e  arn  m 
me  ‘act  that  vhen  tne  t-e  a""  linkage  w-  c~  cos  •  c-s 
t-e  wr.st  s  retracted.  .?  fads  „o  on  tse  1  cer~  ”  -g  me 
arm  'o  oe  quite  compact  ‘or  a  given  wem  -g  e-ve  oce 
ana  ight  m  weight  'cr  a  given  paycad 

T.vc  cj , sti net  mechanical  emoodime-ts  c‘  the  c-'ec 
arm  geometry  nave  gamed  acceptance  m  mdustny 
o-e.  the  actuators  wh'Ch  drive  the  arm  ana  wrist  ,cm;s 
are  mounted  near  me  case  of  the  machine,  at  seme 
distance  from  the  joints  which  they  operate  m  srt.cn 
cesigns,  motors  gear  reducers  and  posi'cn  feedback 
devices  ccated  at  t-e  "sneuider"  rtansm  t  pcwer  to  me 
„  r,s  mrcugh  t-e  effects  of  a  four-par  engage  or 
•-o„gn  oeil-cranks  and  pushrods,  or  by  cnams,  t  -"mg 
certs  cr  seme  other  "tencon"  arrangement,  0-e 
excellent  embodiment  of  this  approach  to  a  jointed  arm 
is  tne  ASEA  IRb  6™  industrial  robot.  Tne 
remote-mounted  drive  approach  has  certa  n 
advantages,  particularly  in  terrestrial  robot  applications 
where  the  requirements  of  the  gravity  field  can  be  a 
predominant  design  factor.  These  advantages  relate  to 
the  disposition  of  the  mass  of  the  joint  drives  off  of  the 
arm  itself,  and  the  consequent  reduction  of  power 
requirements.  Nevertheless,  the  drive  tram  which  is 
employed  to  transmit  power  to  a  remote  joint  itself 
imposes  a  number  of  significant  performance  penalties. 
It  often  restricts  the  range  of  motion  of  the  joints, 
resulting  in  relatively  limited  working  envelopes, 
torroidal  m  shape.  A  long  drivetrain  also  adds 
considerable  inertia,  compliance  and  mechanical 
inaccuracy  between  the  servo  actuator  and  the  joint, 
producing  less  than  optimal  dynamic  response  and 
stability. 

In  a  second  embodiment  of  the  jointed  arm  geometry, 
substantially  all  of  the  joint  drive  actuators  are  mounted 
m  or  directly  adjacent  to  the  joints  which  they  operate 
A  well-known  example  of  a  design  in  which  actuators 
are  mounted  in  the  link  adjecent  to  the  driven  joint  is 
the  Ummation  PUMA™.  This  drive  arrangement 
overcomes  linkage  travel  problems,  increasing  the 
useful  working  envelope  of  the  arm  ..  the  extent  that,  m 
some  models,  envelopes  approach  the  theoretical 
sphere.  Additionally,  true  joint-mounted  direct-drive 
designs  completely  eliminate  the  drivetrain  otherwise 
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'ea-red  to  transmit  sewer  to  me  ,cmt s  ‘rom  remote 
actuators,  significant'y  reducing  criveime  compliance 
and  media  problems  In  *errestr. al  rooot  applications, 
•me  significant  pena.ty  assoc. ated  with  mounting  the 
crives  in  tne  joints  rentes  to  me  additional  we  ght 
carried  m  the  arm  itself.  Wh'e  the  overall  we^gnt  of 
most  .Cint-mounted  drive  arms  s.  n  fact,  'ower  man  tnat 
of  most  remote-enve  systems  of  comparable  dimension 
and  capacity,  tne  ccaton  cf  'mat  ~ass  is  of  great 
concern  n  a  gravity  tie  0.  Terrestrial  joint-mounted 
o've  arms  wnen  are  net  somehow  counterbalanced 
"'av  m  .ize  as  m.^c-  as  303o  of  me  actuator  continuous 
torque  ratings  yust  to  support  ’me  arm  ,n  its  west-case 
oosture.  The  remainder  s  available  to  support  ana 
accelerate  the  payload.  For  space  applications,  me 
overall  weight  of  tne  device  >s  a  critical  issue,  but  the 
way  m  which  the  mass  is  disposed  is  of  less  concern 

From  the  viewpoints  of  spatial  efficiency,  tool-handling 
dexterity,  size,  weight  and  dynamic  performance,  an 
articulated  arm  geometry  which  utilizes  joint-mounted 
actuators  provides  superior  performance  as  compared 
to  alternate  configurations.  The  authors  beiieve  that 
such  designs  should  be  the  leading  candidates  ‘or 
space  operations. 

Jointed  arm  manipulators  require  relatively 
sophisticated  motion  control  systems,  capable  of 
performing  elaborate  and  time-consuming  coordinate 
transformations  to  generate  joint  positions  to  locate  and 
orient  the  tool.  The  complexity  and  cost  of  computer 
control  systems  required  for  accurate  and,  increasingly, 
adaptive  control  of  the  toolpath  in  jointed  arm 
manipulators  have  had  a  significant  influence  on  the 
particular  linkage  geometries  utilized  in  most 
commercial  versions  of  jointed  arms.  Kinematic 
geometries  have  generally  been  adopted  which 
Simplify  the  process  of  transformation  from  joint 
coordinates  to  cartesian  space,  and  back,  in  an  effort  to 
-educe  the  number  of  and  rate  at  which  computations 
must  be  performed.  For  example,  common  jointed  arm 
'mkage  geometries  avoid  rotated  ''off-set”  pitch  joints,  a 
‘eature  which  has  distinct  advantages  in  increasing  the 
useful  workspace  of  the  arm,  but  which  greatly 
complicates  coordinate  transformation  (Figure  1).  By 
mposing  specific  constraints  on  linkage  geometry, 
explicit  mathematical  expressions  (i.e.,  closed  form, 
analytic  expressions)  can  be  obtained  for  the 
coordinate  transformations  In  most  commercial  6  DOF 
arms,  the  control  system  is  simplified  at  the  expense  of 
manipulator  dexterity  and  mechanism  efficiency. 


Articulated  manipulators  which  incorporate  six  joint- 
mounted  drives,  while  more  spatially  and  mechanically 
efficient  than  other  general  purpose  arm  designs,  are 
considerably  less  maneuverable  and  dexterous  than 
biological  analogs  they  would  ideally  emulate,  notably, 
the  human  arm  or  an  octopus  tentacle.  As  previously 
stated,  six  degree  o'  freedom  articulated  arms  function 
like  backhoes  from  a  kinematic  viewpoint,  in  that  the 
arm  linkage  extends  and  retracts  m  a  fixed  plane  which 


Figure  1  Off-set  pitch  joint  kinematics. 
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_  rotated  about  a  vertical  axis  oy  the  base  revoiute 
,,nt  With  most  such  devices,  a  given  location  and 
Orientation  of  the  tool  corresponds  to  a  single  discrete 
-et  of  ;omt  angles  and  an  associated  unique  arm 
configuration  In  a  few  such  devices,  a  given  tool 
ocanon  and  attitude  can  be  achieved  by  two  discrete 
arm  configurations.  An  example  of  a  device  with  'wo 
ocss.ble  arm  configurations  ‘or  a  given  tool  position 
orientation  s  tre  Ummation  PUMA™  In  mat 
oevice,  wnne  the  -evoiute  cmts  representing  the 
'snoulder'  and  "elbow'  pitch  axes  remain  fixed  <n  a 
piane  with  respect  to  one  another,  the  elbow  joint  can 
oe  disposed  either  Up’  or  ’down"  Nevertheless,  if  ‘or 
a  prescribed  position  of  the  tool,  an  obstacle  in  the 
workspace  or  the  workpiece  itself  interferes  with  the 
arm  link  segments,  the  arm  is  not  capable  of  reaching 
the  goalpomt  without  collision  (Figure  2],  Unlike  the 
Human  arm.  a  six  axis  jointed  arm  manipulator  does  not 
"ave  sufficient  degrees  of  freedom  to  reconfigure  itself 
'o  reach  around  an  arbitrarily  placed  ob|ect  in  the 
workspace.  The  human  arm  is  considered  by  the 
authors  to  have  seven  degrees  of  ‘reedom  from 
snouider  to  wrist,  providing  a  range  of  possible  eibow 
attitudes  and  resulting  arm  configurations  'or  a  given 
tool  position  and  orientation 


6  OOf  Articulates  Arm 


F'gure  2  Maneuverability  of  a  6  DOF  vs.  7  DOF 
articulated  arm  'Robotics  Research  Corp 
K-2107  Dexterous  Manipulator  illustrated). 


Many  space  servicing  tasks  wll  -ecu. re  me  ce«te'  u  c 
a  ncman  arm.  The  authors  oeneve  mat  sever,  deg'ees 
of  freedom  should  be  considered  me  m 
incorporated  in  the  tool-handling  arms  ot  a  ge"era 
ourpose  manipulation  system,  such  as  the  NASA  F  gm 
Teierobotic  Servicer  FTS)  The  authors  ‘e""1 
art, coiated  mechanical  arms  having  seven  or  mo 'e 
cegrees  of  'reedom  m  series,  "dexterous  man  pu  ators ' 

Additional  degrees  of  freedom  may  prove  extreme^ 
useful,  as  wed.  The  addition  cf  a  three  aegree  ot 
,reedom  "torso",  upon  which  is  mounted  two  seven 
degree  of  freedom  arms,  may  be  an  ideal  general 
purpose  manipulator  system  for  prospective  FTS 
assembly  and  servicing  tasks. 

Some  tasks  to  be  executed  may  require  more  than 
seven  degrees  of  freedom  in  senes  in  one  articulated 
arm.  The  internal  inspection  of  complex  spacecraft 
assembles  with  a  hand-mounted  camera,  ‘or  examce 
could  -equire  the  maneuverability  of  a  snake  Whre 
such  a  task  may  seem  unlikely  at  this  point,  anq  is 
certainly  an  unreasonable  requirement  for  a  oase  ~e 
design  for  space  manipulators,  reconfiguraoie 
manipulator  system  designs  which  permit  the  'eacy 
addition  of  one  or  more  joints  redundant  joints  or  ,we 
subtraction  of  unnecessary  joints  where  ire 
manipulator  is  dedicated  to  a  single,  constrained  task) 
are  seen  by  the  authors  to  offer  significant  advantages 
over  the  long  run. 

The  addition  of  one  or  more  "redundant"  joints  m  a 
manipulator  has  a  number  of  significant  benefits 
beyond  flexibility  or  maneuverability.  In  the  same  way 
that  an  extra  degree  of  freedom  provides  means  to 
reconfigure  the  arm  to  reach  around  an  obstacle,  the 
arm  can  be  automatically  reconfigured  to  dispose  joints 
in  a  way  which  distributes  torque  or  velocity 
requirements  among  the  axes  in  an  equitable  manner 
A  human  reconfigures  his  arm  (and  torso  and  legs)  m 
the  process  of  lifting  a  heavy  object  to  keep  the  forces 
and  moments  applied  at  each  and  every  joint 
appropriately  distributed  and  minimized.  A  human 
uses  the  redundancy  in  his  arm/torso/legs  to  shift  his 
center  of  gravity  and  to  maximize  his  "leverage".  In  a 
six  degree  of  freedom  jointed  arm  manipulator,  no  such 
reconfiguration  of  joints  and  redistnbution  of  forces  is 
possible.  Thus,  the  mechanism's  capacity  to  apply  a 
particular  force  and  torque  vector  at  the  tool  may  often 
be  unreasonably  limited  because  only  a  few  joints  are 
contributing  lo  the  exercise  ^  Conolis  forces,  which 
could  be  employed  to  advantage,  go  entirely 
uncontrolled. 

Equally  important,  intrinsic  to  all  jointed  arm 
manipulators  is  a  condition  known  as  a  "singularity"  4 
Singularities  are  regions  within  the  working  envelope 
of  the  linkage  in  which,  either  1)  no  individual  joint  m 
the  linkage  lies  in  a  position  and  an  orientation  in 
which  it  can  contribute  significantly  to  the  commanded 
toolpoint  direction  within  its  physical  limits  of  travel  and 
acceleration;  or  2)  more  than  one  joint  in  the  linkage, 
independently,  can  execute  the  commanded  motion 
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T-e  er'ect  s  me  same  T-e  arm  image  "as  lost  ore  or 
—C'e  oegrees  of  ‘reedcm  Arm  oenavior  signif'canriy 
aegrades  w^en  toe  tococmt  s  ^  toe  verity  of  a 
s  -iguianry.  smce  me  mkage  s  u.o ao  e  at  Mat  instant  to 
""a.ntam  me  commaoced  toe  pom*  cosher  3^0. or 
ooentaticn  T-e  equations  tvp  C3!'y  -sea  to  ccnt-c; 
•"0!  on  3t  Me  toe  DC  n*  o‘  3  C'"'ea  arm  "3ve  “0 
■"  at"  e~a:  ca,  S0:wt*0"  it  a  ;  y'y  W~-  e 

encounte'  ”5  o-e  ~ay  oe  e  e-ema"/  oe”  ng  cot  can 
ce  c  -‘  t  <  -e“3'  c  -ecu-oa-c*  cm.  ces  ~ea-s  to 
-a~c  e  me  coo  e~  c*  ;  -a-  a' t  03  ;n  .3  *.  rs^at  c a  y 
'  e  c  _  -  c  a  -  *  ~2"C-.3to'  -"C'0  fan  one  set  of  ;cm? 
a- g  es  can  0lJ  a~- co  .ee  *o  atta  "  a  conv”a"ced  tool 
cos  con  The  aufe's  p-  r-c  cany  Fa-reit)  ard  H&vard  l. 
Vo  a  nave  devised  computationally  efficient  a'gorithms 
oy  'ivr.cn  fe  axes  n  a  kinematically  redundant 
~an  ou.ator  can  oe  coohgurea  to  contriDute  to  any 
arpitrary  commanded  tool  dnect.onai  and  'otationai 
zeetcr.  s"aD‘irg  'me  arm  to  exit  a  singularity  in  a 
control  ed  fashion  T-  s  algorithm  may  have 
application  >n  general  purpose,  kmemahca'ly 
■edundant  manipulator  systems  for  space  operations 

Ssecai  Purpose  Maroeato's 

In  addition  to  the  four  basic  types  of  general  ourpose 
manipulator  geometry  reviewed  above,  each  of  whicn 
provides  six  degrees  of  freedom  at  the  tool,  many  other 
rkage  arrangements  providing  fewer  controlled  axes 
-ave  been  devised  ‘or  special  applications.  In  the 
design  of  most  such  special  puroose  arms,  an  effort  is 
made  to  empioy  a  linkage  with  the  minimum  number  of 
driven  joints  to  perform  a  unique  tool  trajectory  or  class 
of  trajectories  Significant  cost  savings  result  from  such 
an  effort  through  the  reduction  m  the  number  and  often 
the  size  of  structural  components,  joint  actuators,  power 
supplies  and  servo-control  hardware. 

Soeciai  purpose,  limited  degree  of  freedom 
manipulators  have  found  wide  use  in  industrial 
applications  where  a  device  is  dedicated  to  one 
installation  and  task  for  its  useful  life.  The  application 
must  be  well  defined  and  the  associated  trajectories 
must  Oe  fully  constrained  for  these  devices  to  perform 
the  function.  Nevertheless,  the  majority  of  factory 
automation  applications  are  today  being  accomplished 
witn  manipulators  which  incorporate  fewer  than  six 
degrees  of  freedom  (five  axes  is  the  most  prevalent 
geometry).  Special  purpose  geometries  are  developed 
oy  careful  selection  of  appropriate  joint  axes,  link 
engths  and  base  location  with  respect  to  the 
workspace.  Common  examples  of  this  specialization  in 
industry  include  four  axis  lathe  loading  devices,  and 
five  axis  arc  welding  robots. 

Such  specialization  is  natural,  of  course,  and  a  similar 
trend  can  oe  anticipated  m  space  robotics  applications 
as  time  goes  by.  It  may  be  determined,  for  example, 
that  four  degrees  of  freedom  are  fully  sufficient  to 
perform  silicon  furnace  servicing  operations  in  one  bay 
of  the  Space  Station  Possibly  the  correct  three  axis 
arm  could  perform  many  repetitive  laboratory 


excer  ~er's  Sc^e  com-actcr  st-c  es  suggest  •“a'  — e 
process  of  rep-acmg  GPUs  or  me  ous  c  a  sate  -e  - 
geosynchronous  cro-t  can  'eac.-y  oe  aco~o  s~ec  cy  a 
‘./e  ax  s  manipulator  car- eg  by  me  S'O'a 
Mareuver  rg  Vehicle  iCMV' 

T-e  authors  oe  eve  ma:  a  wee  -a^ge  of  c  f'erent  v-- 
s.zes  and  cor*  g„-at  ons  w.  -  t  mate.y  ce  ceie'  —  "eo 
to  be  appropriate  ‘or  the  various  roDotic  ana  teiercoct  c 
space  servicing  coerat'cr.s  T"ese  "-ay  range  n  erg:- 
‘-cm  arms  wan  tn«  s.ze  and  oay  oad  of  me  Remote 
Manipulator  System,  cewn  to  arms  of  .ess  man  r_-an 
sca.e  Tuey  m  got  cea  y  "cc'ocrate  as  few  as  mree 
axes  ‘cr  certain  applicat  ors,  and  as  many  as  twenty  ‘or 
ethers.  Each  of  these  "’a"  opiates  c;u  a  oe 
custom-designed  ‘or  its  tas«  Too  autrc's  oe  .eve  mat 
a  standard  mcdu'ar  '"ar.pu.ator  system  w-  c- 
‘acii. tales  the  assemo  y  of  a  orcaa  family  of  ar~s  ‘'em  a 
set  of  common  jomt-drve  mocuies  ccu.d  provee  a  org 
term  solution  to  this  problem  of  $p«>mai,  731,00 


Mofon  Control  Architecture 

It  ,s  worth  noting  that  the  type  of  arm  motion  cortrci 
computer  system  that  is  used  to  perform  toc^omf 
trajectory  control,  coordinate  transformation  and 
servo-control,  regardless  of  arm  configuration,  can  oe  a 
relatively  independent  issue  from  what  higher  level 
control  system  is  employed  to  generate  arm 
commands.  A  properly  segmented  hierarchical  control 
system,  as  suggested  by  James  S.  Aibus.  Anthony  J 
Barbera  and  Mary  Lou  Fitzgerald,  decouples  the  arm 
motion  control  from  higher  level  functions  5.  6  jn  an 
autonomous,  sensory-interactive  robot  control  system, 
higher  levels  provide  the  user  programming  interface, 
accomplish  task  decomposition,  break  move 
statements  into  primitives  and  process  information  from 
global  sensor  systems,  such  as  cameras.  The  arm 
motion  controller  accepts  goaipoints  in  any  arbitrary 
coordinate  frame  of  reference  and  proceeds  to  execute 
the  commands  and  report  status.  The  motion  controller 
concentrates  on  its  local  sensors,  measuring  joint 
position,  velocity,  forces  and  torques,  internal 
temperature,  and  perhaps  utilizes  arm-mounted 
proximity  sensors  for  local  obstacle  avoidance.7  The 
higher  level  controller  thus  directs  the  arm  as  a 
peripheral.  The  "host’  controller  does  not  necessarily 
have  to  know  what  arm  it  is  directing,  or  its  unique 
geometry.  This  architecture  is  of  particular  significance 
in  the  design  of  robotic  systems  for  prospective  space 
operations  for  at  least  two  reasons.  First,  with  a 
hierarchical  structure,  any  particular  arm  can  be  driven 
as  a  peripheral  either  by  an  autonomous  robotic  control 
system,  or  by  some  man-in-loop,  teleoperated  master 
controller.  The  arm  does  not  recognize  the  difference 
This  has  important  implications  for  the  feasibility  of  a 
standardized  manipulator  system  for  space  operations. 
Secondly,  the  hierarchical  control  system  might  be 
viewed  as  a  structured,  "from  the  ground  up"  approach 
to  implementing  artificial  intelligence.  Each 
hierarchical  level  operates  relatively  independently  as 
a  closed-loop  controller.  In  this  context,  a  hierarchical 
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"3CQ!iC  control  architecture  provoes  -eacy  •’•ea-g  *;> 
system  expansion  ana  enhancement  as  mproved 
sensors  and  algorithms  are  aevisea  ‘or  any  given  evei 


nterface.  facilitating  ,se  of  a  sra-ca'd  set  ;■* 
manipulation  Hardware  n  a  w:ce  var-ety  of 
autonomous  and  telecperated  systems 


CorclLSion-  Opportunities  ‘or  a  S:a^card  Mgfl'-uai 
du  ator  System  ‘or  Space  Ober3!  pnS 
S  M  M  S ) 


i"  cor'c'^s.on.  :-e  authors  oeiieve  •mat  a  ccnsiderapie 
cocv  of  ev.cence  indicates  t  iat  the  appropriate 
case  re  con*. gurat.cn  ‘or  the  dexterous  mecnamcai 
a 'm  systems  to  be  used  ‘or  toot -Handling  in  soace 
servic.ng  tasks,  wnether  operating  under  "autonomous" 
computer  program  control  or  telecperated,  is  an 
articulated,  seven  degree  of  freedom  device  having 
modular,  jo.nt-mounted  drives  and  a  standardized, 
deal  motion  controller. 

Furthermore,  the  authors  propose  that  NASA  and  the 
USAF.  m  concert,  consider  developing  and  adopting  a 
Standard  Modular  Manipulator  System  (SMMS)  for  all 
rcootic  and  telerobotic  assembly,  retrieval,  experiment 
support  and  servicing  applications  on  the  Space 
Station  and  for  on-oroit  remote  satellite  servicing. 

SMMS  is  envisioned  by  the  authors  as  a  family  of 
articulated  manipulators  ranging  in  Size  from  less  than 
tnree  feet  to  over  fifty  feet  in  length,  incorporating  as 
many  joints  as  necessary  for  a  given  application,  each 
manipulator  being  an  assembly  of  unitized  joint  drive 
modules  of  common  design  (Figure  3). 

Lightweight,  interchangeable  joint  drive  modules,  each 
navmg  as  few  as  two  moving  parts,  could  be  designed 
to  provide  a  set  of  standard  revolute  "roll’  and  "pitch* 
joints  with  output  torque  capacities  ranging  from  40  to 
40,000  in. -lbs.,  in  appropriate  steps.  Individual  joint 
drive  modules  would  carry  across  the  family  of  arms. 
For  example,  a  2,500  In. -lb,  pitch  module  which  serves 
as  the  elbow  in  a  20  foot  long  manipulator  would  also 
be  used  as  the  wrist  pitch  module  in  a  fifty  foot  arm  and 
the  shoulder  pitch  module  in  a  five  foot  arm.  In  addition 
to  their  use  as  arms,  these  same  modules  could  be 
assembled  into  configurations  which  serve  as  torsos  for 
muitiarm  devices  and,  indeed,  "legs'  for  certain 
vehicles  operating  on  the  Space  Station  truss  (Figures 
4  and  5). 

The  authors  propose  that  an  advanced  motion 
controller  conforming  to  a  standard  heirarchical 
architecture  be  utilized  for  each  manipulator  assembly. 
Each  motion  controller  would  have  the  capability  to  use 
an  array  of  local  kinesthetic  sensors  to  perform  the 
"brainstem"  functions  associated  with  the  arm.  In 
general  purpose  assemblies,  incorporating  seven  or 
more  degrees  of  freedom,  the  motion  control  functions 
would  include  coordinated  joint  control,  real  time, 
sensor-based  arm  obstacle  avoidance,  and 
optimization  of  arm  posture  for  leverage. 

With  this  architecture,  SMMS  arms  could  operate  as 
intelligent  peripherals  to  any  task-appropriate  host 
computer  through  a  common  hardware/software 
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Figure  3.  Robotics  Research  Corp.  concept  for  a 
Standard  Modular  Manipulator  System 
(SMMS)  for  space  operations. 
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F;g ure  4.  RAMA/ATS  concept  for  a  legged  vehicle  for 
Space  Station  servicing  employing  SMMS 
modular  manipulators. 


Figure  5.  Robotics  Research  Carp,  concept  for  a  17 
DOF  anthropomorphic  manipulator  for 
Space  Station  assembly  and  servicing  (built 
from  SMMS  unitized  joint  drive  modules). 


The  authors  anticipate  that  the  standardization  of 
mechanical  designs  and  control  interfaces  proposed  in 
SMMS  would  substantially  enhance  manipulator 
rystem  reliability,  would  minimize  development  and 
production  costs,  and  would  readily  accommodate 
future  growth  in  space  operations  and  changes  in 
mission. 


At  Robotics  Research  Corporation,  m  Milford,  Ohio,  the 
authors  and  their  colleagues  have  developed  and 
demonstrated  a  modular,  articulated  manipulator 
system  which  embodies  many  of  the  characteristics 
required  for  SMMS.  The  'K-Series"  Dexterous 
Manipulator  system,  while  designed  m  scale  and 
details  for  terrestnal  factory  automation  and  laboratory 
use,  may  serve  as  a  reasonable  prototype  for  future 
space  manipulators.  Current  production  models  may 
provide  ready  means  for  terrestrial  demonstration  and 
experimentation  with  kinematically-redundant 
manipulators. 

The  K-Series  Dexterous  Manipulators  are  built  up  from 
a  family  of  unrtized  joint  drive  modules.  Each  module 
includes  a  complete  joint  dnve  mechanism,  feedback 
transducers,  beanngs  and  structure  for  a  degree  of 
freedom.  In  existing  units,  individual  joint  drive 
modules  do  not  contain  the  signal  conditioning,  control 
and  servo  power  electronics.  These  components  are 
housed  m  a  control  cabinet  and  connected  to  the 
modules  with  a  highly  flexible  internal  winng  harness. 
Each  module  is  designed  around  a  particular  size  (and 
thereby  torque  capacity)  harmonic  dnve  reducer  ana 
incorporates  a  DC  servomotor  of  appropriate  power 
output.  This  drive  system  can  be  tailored  to  meet  a 
wide  range  of  application  requirements  by  selecting  an 
appropriate  harmonic  drive  ratio  between  60:1  and 
320:1.  The  modules  are  of  two  types,  configured  as 
"roll*  and  *pitch*-type  joints.  Roll  modules  effect  rotary 
motions  about  the  axis  of  the  module  interface  flanges, 
wh’le  pitch  modules  effect  rotary  motions  about  an  axis 
perpendicular  to  the  normal  vector  of  the  attachment 
flanges.  Modules  are  joined  to  each  other  by  V-Band 
cfamps,  secured  by  a  single  tangent  bolt. 

These  roll  and  pitch-type  modules  are  designed  to  be 
assembled,  in  series,  into  a  variety  of  different  arm 
sizes,  payloads  and  kinematic  configurations. 
Individual  joint  modules  can  be  utilized  across  a  family 
of  arms  (Figure  6).  Two  models  in  the  line  are  shown  in 
accompanying  photographs.  The  K-2107HR  is  a  seven 
foot  long,  seven  axis  manipulator  configured 
specifically  for  applications  which  require  a  light-weight 
tool  or  sensor  to  be  conveyed  about  a  large  working 
envelope  with  exceptional  dexterity  and  speed,  and 
with  the  greatest  possible  positioning  repeatability 
(Figure  7).  Operating  in  a  stable  temperature  state,  the 
K-2107HR  will  repeatedly  position  a  tool  to  within 
4/10,OOOths  of  an  inch.  The  K-1607  is  a  five  foot  long, 
seven  axis  unit  configured  for  general-purpose  factory 
and  laboratory  use  (Figure  8). 

Kinematically-redundant  versions  of  K-Series 
manipulators  (incorporating  seven  or  more  degrees  of 
freedom)  offer  tool-handling  dexterity  which 
approaches  that  of  a  human  arm.  The  seven  degree  of 
freedom  arm  configuration  has,  in  fact,  the  same 
number  and  a  similar  disposition  of  joints  as  a  human 
arm  The  K-Series  manipulator  actually  eclipses 
anthropomorphic  capabilities  in  range  of  motion  for  arm 
roll  joints,  producing  +/-180  or  +/-360  degrees,  where 


the  human  arm  ;cmts  are  tyD'caliy  'i^i'.ea  to  +  -45 
degrees  or  less  The  motion  range  available  in  ’me 
K-Series  pitch  modules  sightly  exceeds 
anthropomorphic,  with  a  fuil  ISO  degree  range.  This 
range  of  travel  >n  pitch  joints  is  possible  through  the 
use  of  an  offset  geometry,  where  the  pitch  joint  ax>s  ;s 
displaced  from  ’he  attacned  roil  axes  ire’er  to  F-gure  t : 

The  disposition  of  modules  wit  hi  n  a  K -Senes 
manipulator  ‘o  lows  the  pattern  of  roll-pitch -roll- 
pitcn-etc.  If  a  manipulator  of  reduced  degree  of 
freeaom  is  des.red,  the  most  reasonable  degeneration 
of  capability  is  to  replace  a  roil  module  position  in  the 
Mnematic  Cham  with  a  static  imk  (i.e..  a  tubular 
structural  element  which  nas  attachment  provisions  ?n 
both  anas  for  p.tch  modules). 

The  generic  actuator  design  utilized  in  all  K-Senes 
modules  consists  of  a  harmonic  drive  located  on  the 
oint  axis,  directly  coupled  to  the  proximal  and  distai 
castings  of  the  module  A  high  performance 
samanum-cobait  DC  servomotor  is  directly  coupled  to 
the  harmonic  drive  wave  generator.  The  flexspime  of 
‘he  harmonic  drive  couples  to  the  structure  through  a 
metal-to-metal  slip  clutch  and  torque  transducer.  The 
s; i o  clutch  is  generally  adjusted  to  slip  at  a  torque 
greater  than  required  for  full  machine  performance,  but 
ess  than  wouid  damage  the  drive  or  other  manipulator 
components.  An  independent,  high  precision 
instrument  gear  system  drives  a  brushless  resolver 
i to  provide  joint  position  and 
non-quamtized  velocity  feedback  by  means  of  an 
advanced  R-to-D  chip  By  this  arrangement,  the  servo 
system  has  applied  actuator  torque,  joint  velocity  and 
joint  position  feedback  available  to  provide  controlled 
joint  or  axis  motion. 

The  utilization  of  the  harmonic  drive  reducer  m  this 
fashion  is  both  effective  and  problematical.  The 
harmonic  drive  is  unrivaled  for  compact,  light, 
backlash-free  torque  multiplication.  It  also  exhibits  a 
two-per-input-revolution  transmission  error  which 
excites  the  inevitable  resonance  resulting  from  the 
inherent  reducer  compliance.  This  often-suggested 
arrangement  of  actuator  system  is  typically 
handicapped  by  an  obvious  and  objectionable 
resonance  in  op^ahon.  The  resonance  phenomenon 
intrinsic  to  direct-drive  arrangements  of  harmonic 
drives  in  robot  arms  has  prevented  the  widespread 
application  of  this  otherwise  attractive  actuator 
package.  K-Series  manipulators  utilize  a  servo-control 
approach  which  overcomes  this  problem  and  provides 
important  attributes  for  advanced  actuator  control  and 
improved  manipulator  performance.  The  conventional 
approach  in  robot  arms  is  to  control  velocity  and 
position  of  the  motor  shaft,  while  assuming  that  the 
transmission  elements  are  nearly  ideal  in  their 
translation  of  shaft  motion  into  manipulator  motion.  The 
approach  taken  by  on  of  the  authors  (Thompson)  and 
colleague  Paul  H.  Eismann  in  K-Series  servo  dnves  is 
to  treat  the  motor  and  harmonic  drive  as  a  torque 
producer  The  control  feedback  parameters  are  all 
measured  at  the  interface  between  proximal  and  distal 
joint  elements.  The  joint  position  commands  joint 


>-e  c c •  f y  wr  c"’  zcmn-anqs  aco  ec  zr  .  v;-  • :  • : .  - 
Tne  '■rermcst  iccp  is  thus  a  .'c-'cut?  crc  :.\z ~ 
bandwidth  which  encompasses  me  ''c"~a  'e _v: 
‘requency  range  of  the  actuator  package  T-.s  ’c'o-e 
■oop  can  be  visualized  in  several  ways  Ore  v  ew  - 
mat  4  operates  as  an  active  damoer  ‘cr  me  m  c' 
resonance  Another  view  is  that  t  prcx-des 
ces.gner  an  opportunity,  withm  the  acn  evao  e  ::c 
bandwidth,  to  funaamentally  re-write  me  const  me”: 
equation  governing  the  behavior  of  the  drive  as  a 
torsional  spring  between  two  inertias.  Th?  cc"'rc  sod 
actually  functions  to  position  the  motor  red  a.  -cwever 
•  t  must,  to  cause  the  drive  to  prov-ce  me  oesreo 
app.'ea  ;cmt  toraue 

As  previously  mentioned,  the  forage-  cco  s ,  e : - ~ 
provides  significant  advantages  beyond  e!’m  "a:  rg  me 
narmomc  drive  resonant  response  to  prcvice  s~os:~ 
motion.  A  significant  trend  in  research  cn  man!0-  a-:- 
control  s  to  command  axis  torques  to  O'z ■  ce 
h.gn-bandwidth  tool  force  control  or  ’  —ceds-’ce 
control"  In  K-Series  arms  the  fastest  'coo  a:  wc'< 
the  servo  control  u/stem  ;s  the  tc'que  ooo  ; 
innermost  loop  can  remain  in  operation  wm  e  ~z-ze 
switching.  (Joint  torque  transducers  are  s 

elements  to  each  ,omt  in  the  machme1  T~e  •?-:  ,e 
'ocp  also  encompasses  and  compensates  ‘cr  ~:-:r 
motor  seal  and  drive  friction.  Viewed  from  outs.ce  ‘me 
ocp.  it  imparts  lo  the  system  most  of  the  a::r.o_:es  c* 
best  direct-drive  manipulators,  while  avoiding  me 
relatively  poor  torque-to-mass  ratios  of  the  direct-cve 
motors. 

The  K-Series  module  system  utilizes  an  exoske  e‘on 
structural  approach.  The  exoskeleton  struc‘„re 
provides  favorable  structural  dynamics  of  the  overaii 
manipulator,  with  low  overall  weight.  It  also  provices  a 
strong,  durable  and  clean  exterior,  enclosing  all  w-rmg 
and  actuator  componentry. 

A  standard  "black-box’  motion  controller  “as  seen 
developed  to  control  any  arm  configuration  that  can  ce 
assembled  form  K-Series  modules.  This  acceDts 
position  commands  from  any  host,  in  gioDai 
coordinates  or  some  arbitrary  frame  of  reference,  and 
performs  the  trajectory  control  and  coordinate 
transformations  necessary  to  effect  the  commanded 
toolpomt  motion.  Such  global  goalpomts  can  be 
up-dated  on  a  50-100  millisecond  basis,  providing 
sensory-interactive  control  capability.  The  architecture 
of  this  motion  control  is  illustrated  in  the  accompanying 
figure  (Figure  9). 

The  alaorithm  devised  by  the  authors  and  Void  to 
accomplish  coordinate  transformations  for  7+  degree  of 
freedom  manipulator  configurations  accomodates 
singularities  in  the  workspace,  provides  means  to 
employ  the  additional  degrees  of  freedom  to  avoid 
obstacles,  and  to  distribute  in  a  reasonable  manner  the 
actuator  torque  requirements  amongst  the  joints.  The 
algorithm  can  be  configured  to  operate  any  number  of 
manipulator  joints.  The  more  degrees  of  freedom  in  the 
manipulator,  the  more  graceful  the  arm  motion 
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-  g„re  3  Rcccvcs  Research  Coro.  K-Ser:es  Motion 
C:"!'c  3 /stem  Architecture 


T-e  k-sa'es  Dexterous  Manipulators  should  provide 
jss’Oprate  test-beds  for  terrestrial  research  programs 
-eat'cg  '3  ‘-ture  space  manipulation  systems,  e.g  .  in 
■~e  areas  of  adaptive  'orce/position  control,  advanced 
ooerat  on  techniques  and  multi-arm  coordination 


Co'e  '3  Dual-arm  arrangement  of  K-1607  Dexterous 
■Mampuiatcs 


fee  K-Senes  Manipulator  System  is  seen  Dy  me 
authors  to  embody  the  fundamental  mechanism  ana 
control  technology  required  to  implement  dexterous 
tool -handling  manipulators  for  space  servicing 
applications,  including  the  overall  SMMS  concept  ror  a 
‘ami'y  of  modular  arms.  The  developments  required  for 
a  soace  version  appear  relatively  few  and 
straightforward 

T~e  high  vacuum  environment  will  necessitate  the  use 
o‘  orusnless  motors.  A  brusnless  system  otters 
Significant  improvement  in  arm  thermal  management 
and  reduces  the  electromechanical  system  to  an 
extremely  simple,  reliable  pacKage.  Brushless  motors 
•v.:i  improve  system  reliaoiiity  by  providing  redundant 
e  ectronic  systems.  Their  only  disadvantage  is  that 
fney  require  additional  interface  wiring  between  motor 
posit, on  transducers  and  the  power  electronics. 

Harmonic  drive  lubrication  needs  to  be  carefully 
consicered.  with  respect  to  outgassing.  vacuum 
staointy  and  general  longevity.  The  narmomc  drive  "as 
been  used  extensively  in  the  space  program,  and  t e re 
s  no  shortage  of  experience  with  this  issue 

Thermal  management  of  the  manipulator  needs  to  be 
addressed  to  assure  acceptable  power  consumption 
and  operational  reliability.  Possibilities  exist  to  achieve 
thermal  control  by  active  configuration  of  insulation, 
reflectivity,  and  radiation  of  the  manipulator  itself.  Other 
possibilities  include  utilizing  heat  pipe  technology  and 
insulation  to  control  the  thermal  state  of  the  manipulator 
as  an  extension  of  (he  thermal  environment  of  the 
spacecraft  itself.  8oth  of  these  approaches  could  be 
augmented  by  phase-change  thermal  storage  sysiems 
to  extend  the  thermal  time  constant  of  individual 
actuator  systems. 

Individual  joint  drive  modules  need  to  be  designed  as 
Orbital  Replacemer:  Units  (ORUs).  In  order  to  repair  a 
manipulator  which  has  malfunctioned  by  exchanging 
modules  (possibly  using  another  manipulator), 
everything  which  makes  the  module  operate  must  be 
contained  within  the  module,  or  ORU.  itself  Taking  the 
ORU  concept  and  the  overall  system  mass  and 
reliability  goals  into  consideration,  one  clear  strategy 
for  design  becomes  integrating  the  actuator, 
transducers,  signal  conditioning,  analog  and  digital 
control  electronics,  brushless  motor  power  and 
commutation  electronics  and  a  command/ 
communication  interlace  into  one  ORU.  Each  ORU 
might  include  one  or  two  axes  of  motion,  depending  on 
replacement  strategy  and  reliability  analysis. 
Consolidation  of  these  system  elements  into  a  ORU 
package  will  minimize  the  complexity  of  the  flexible 
wiring  harness  and  many  of  the  potential  failure  modes 
associated  with  the  harness  Instead,  a  power  bus  and 
a  communication  bus  would  be  incorporated,  each 
tr, pie-redundant.  Communications  could  be  through 
optical  fiber  or  utilize  powerful  bus  communication 
techniques.  All  that  wouid  remain  of  the  off-board 
control  box  is  a  power  bus  controller  and  motion  control 
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computer  intern  space  manipulators  can  certai;  y  3*? 
constructed  snort  of  this  level  of  integration,  but  tnere 
are  significant  re  ao  :.ty  advantages  to  be  gamed 

Preurrmary  designs  of  ORU  modular  manipulators 
currently  ^ncerway  at  Roootics  Research  Corporation 
nocate  ve-y  'avorab'.e  characteristics  may  be  achieved 
,v;th  re soec:  -o  arm  s,ze.  mass  and  reliability. 
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NASA  ".is  ^mrvir  xe.t  ‘-n  i  ?  '  ■*  million  s?a.;»> 
research  an  a  : -c  r.no  1  ogY  program  jver  rive 
■-■(Mrs  c o  begirt  c  )  restore  the  iger.  cv's 
technical  strength.  The  program,  railed  :ne 
:  7 ;  v  1  L  Space  Technology  Initiative,  will 
c:  t  r  i  hiite  t  ■  ■>  the  rechr.ol  >gy  base  that  will 
"■^Ip  reestablish  the  .S,  ;ivil  space  program 

-  \  o  i  j  s  i 1 1  _>  n  ■'  r  w  rid  leadership.  -1ST  \ 

:  :c  -ides  re*se<r."h  :  :*».;nm>l  •  gie<:  :  •>  enable 

-r:i  *ient,  reliable  access  t  ■>  and  '.rerat  inns 
!:•  spa^.e  ,  ir.d  :  nr  technologies  s-.ipp.nr  t  i  ng 
*<  •  :  r*  :u:  e  ®  i  s  s  i  ms. 

r  r  ms  ;•  < .  r  t  a :  1  -■  n  r  « •' k  n  • '  1  '•  g :  •*  s  i  n .- 1:  i  e 
r  **-iSa  hi  e  ,  h  :  gh-perf  irmarme  engines  :  .  r 
ext-ieneratl  -n  vehicles,  ‘■‘Ouster  technology 
opt  1  ;ns ,  ir.d  an  aerobraxing  iiight 
experiment.  Kart h-orbl t i ng  operations 
l^'l’ide  in  expanded  effort  In  automation  and 

-  'boti  's  and  high  apacitv  power.  Science 
*  e  v-  n o 1  i  [  p  s  i  n  ■..  1  jd  e  a  1  a  r  it  e  flexible 

s  t  ru>*  t  .res  md  'or. t  r  ;  1  program,  sensor 
ievi :es  and  high  rate,  high  capacity  iata 
s  vs  terns.  .'ST  I  is  i  means  to  many  ends. 

hist  } r 1 r a  1  1  v  ,  technology  has  been  a 
~  -me  rstone  of  our  leadership  in 
space  i:id  a  ’«ev  factor  to  C.S.  industrial 
■sipet  i  t  i  veness  .  '1ST I  is  a  rlrsc  step  to 
maintaining  that  trend  and  will  be 
:  mr  1  emer  r  ed  in  ;i-se  cooperation  with 
In  hist rv  and  through  expanded  involvement  of 
.riverside*. 

BACKGROUND 


For  twencv-elght  years  NASA  has  pushed  at 
• ke  at  ting  edge  of  space  technology  and  at 
The  frontiers  of  space.  But  now  the  Nation 
is  at  a  decision  point  for  a  renewed 
indication  - o  the  r  l  v  1 1  space  program  --  a 
new  beginning.  Why7  Because  the  once-held 
'i .  position  of  dominance  in  space 
'^-hnologv  has  been  seriouslv  challenged. 

"ir  direction  as  an  agency,  then,  is  to 
reconstitute  the  Nation’s  abllltv  to  supply 
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-pice  * r  insport  it  I  on  services  to, it  will 
preclude  interruptions  l.n  T^e  future, 
md  to  strengthen  tue  technology  K  a  s  e  with 
•ptlons  tKat  :an  sustain  leadership  in  space. 
It  will  take  world-class  facilities,  i  p  d  i  •  1 1  e  i 
pursuit  of  science  and  advanced  t e  : h n o  1  J g i e  ->  , 
and,  more  important  lv,  developing  md  n  .t  :rt  '.T 
the  human  resources  capable  c f  meeting  t v >- 
‘ha  1  lenges  and  id  ventures  of  3ur  futur*-* 
endeavors.  This  challenge  requires  ho:  -  .r 
energy  and  monev.  It  requires  new 
technologies  that  me  r  g  e  the  safety  lessens 
expounded  bv  the  Presidential  Commission  n 
the  Challenger  Accident  and  the  goals  if 
-.-xpanded  future  space  activity  set  fort'-  nv 
rKe  National  Commission  on  Cpace. 

DECT  I N I  NG  INVESTMENTS  AND  CDMPrlT  III  N 


Tb.e  National  Commission  on  Space,  in 
recognition  of  the  need  to  substantially 
increase  the  national  investment  i  .i  space 
technology,  recommended  "A  threefold  growth  in 
NASA’s  base  technology  budget....’  This 
three  times"  budget  should  be  met  in  rive 
years,  for,  no  matter  bow  clear  a  vlsio-  we 
have  of  our  long  range  needs,  we  cannot  move 
forward  agreasively  in  all  necessary 
disciplines  without  a  reversal  of  the  budget 
trend.  Historically,  technology  has  beer,  a 
cornerstone  of  our  leadership  in  space  and  a 
ev  factor  to  ’J .  S .  industrial  competitiveness. 
After  peak  Investment  in  the  early  i960**,  the 
NASA  Space  keoearch  and  Technology  budget  fell 
precipitously  so  that  within  ten  years  we 
dropped  to  a  relatively  constant  Investment  pf 
pnly  one  fifth  to  one  sixth  of  what  It  had 
been  when  the  nation  made  its  commitment  to 
leadership  In  space.  Our  aerospace  industrv, 
the  only  remaining  high-technology  segment  of 
the  U.S.  economy  with  a  positive  balance  of 
trade,  is  facing  increased  competition  and  the 
technology  base  now  contains  gaps  and  a  budget 
Insufficient  to  close  them.  During  this  same 
period,  investments  In  France,  Ttaly,  Germany. 
Japan,  China,  the  Soviet  Union  and  other 
countries  have  Increased  greatly.  What  was 
once  an  unquestioned  area  of  L’.S.  preeminence 
--  the  civil  9pace  program  —  Is  now 
increasingly  a  field  with  shared  leadership. 

TECHNOLOGY  READINESS 


So  many  Innovative  and  exciting 
technological  opportunities  face  us  that 
selecting  them  becomes  a  significant  challenge 
in  ltseif.  But,  the  question  remafns  as  to 
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;  i.-{  f.e  t ev  h:o  l  ?gy  program  r  nr  reusable 
:".-s  t r  it  s bn ild  have  supported 
.  .. :  j  1  :  gv  to  provide  options  to  the 

::  D'^et  to  tor  has  not  been 
i  f  t  ed  .  .  .  . :  l,e  filament  wound  solid  rocket 
r  • as*  lid  not  have  i  basic  engineering 
i  -ase  in  hand  before  iovelopment  was 
: 1  1 1 ed . . . . we  rind  ourselves  dependent  in 
*  s  as  i  power  source  with  ,o  suitable  back 
...we  rice  an  information  lag  and  a  data 
t  Sr.  AS  AT  operated  tor  only  3  months 
it  took  eight  years  to  .analyze  the  data), 
the  Space  Station  era,  the  data  generated 
1  exceed  10  terabits  per  day,  greater  than 
written  information  in  the  Library  of 
gress, . . . The  S P- 1  '’0  space  nuclear  power 
gram  had  to  take  a  conservative  approach 
the  thermal  conversion  system  because  of 
tceptable  risk  with  more  efficient 
version  svs t eras . , . . the  technology  is  not 
dy  for  the  developement  of  new  launch 
1  ’  es  .  .  .  .  spacecraf  t  payload  capacity  is 
e'essarily  limited  because  we  haven’t 
sloped  the  technology  to  lower  the  power, 
pulsion,  and  structural  mass 
■.  t  ion ...  .there  are  parts  of  the 
crrum  we  can't  exploit  for  remote  sensing 
a use  we  haven't  done  the  instrument 
e’.apment  research  .  .  .  .  and  ,  human 
; oration  is  paced  by  our  limited 
♦^standing  of  the  effects  of  space  on  the 
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~he  NCOS  recommendations  are  'uvgru.us  with 
two  other  studies  supporting  the  need  :;r 
increased  research  and  technology  investments 
in  space.  A  three-year  ''pace  S-'ience  ?>oari 
studv,  'Tventv-vear  Forecast  of  Space 
Science,  i  99b-20 1  5> ,  "  provides  a  ligi'ai 
extension  of  the  Nation’s  current  science 
program  with  a  balance  between  as : r * pr vs i c s  . 
planetary  exploration,  earth  so:  .*roe,  and  life 
science.  The  second,  a  ‘Dint  •  d.  NASA  studv. 
Identifies  new  space  :  ransporta  t  i  .'‘“Ki~le 
options  required  to  meet  national  ne*1  is  i  .n 
response  to  a  National  Security  Studv 
Directive  on  space  t r a ns portae  ion .  In  total, 
three  major  studies,  the  Challenger  tradgedv, 
and  foreign  competition  are  transcribed  into  a 
bold  commitment  to  the  future,  *  renewed 
commitment  to  safety,  and  the  restoring  „>f  our 
technical  capability. 


A  FIRST  STF.P 


To  begin  the  task  of  strengthening  our 
space  technology  base,  NASA  has  developed  an 
immediate  first  step  for  FY88  to  be  fallowed 
by  a  second  initiative  that  will  enable  bolder 
missions.  They  are.  The  Civil  Space 
Technology  Initiative  (CSTI)  and  Project 
Pathf  inder . 


Tr.in-^ por  t a  c  i  a r 


CSTI  i  ti'  lodes  research  in  t 'M/hno  1  og  i  es  r  >r 
sir-*  and  p  r  '  :  o :  t- n c  k  joss  r  n  spice,  !  ear"'. 

rSiri  -i  T  ip*-ra:  i-ms  ,  tnd  r ■  > r 
^ipp-^r:  ing  t  i.-r.no  Louies  se*?  ri.-;,  ['•  '  n 

-  r  ir.soort.it  ion  ,  !  i  * :  n  *.-  vi  vehicle  nropul^i-n 
..ill  S‘  i  i  ->*  j  -i*-  reusible,  '".mh-p-rr  .r^anc-1, 
v. v  i r  i^en  i .:  i  '  ••>:  Hv.ir •.wir1'  n  onlines  r  )r 
'vxr  - g  e  oe  r  a :  ion  ea r : h-t o-or bi :  vehicles.  A 
"pw  booster  technology  program  will  develop 
ilternatives  to  the  solid  rocket  motor 
tor  future  transportation.  Technology 
development  will  be  followed  by  a  series  of 
tiring  tests  to  verify  performance 
predictions  of  pressure-fed  liquid  rocket 
propulsion  and  large-scale  hybrid  systems. 

An  aerobraking  flight  experiment  will 
lemons t rate  how  to  use  the  atmosphere  to  slow 
in  orbital  transfer  vehicle  returning  to 
arth  orbit  at  high  velocities.  Application 
of  the  aerobrake  concept  to  the  077  has  the 
n't^ntial  fir  increasing  the  payload 
delivered  to  geosync nronous  orhit  and  return 
by  as  much  as  a  factor  of  two.  In  the  flight 
experiment,  a  highly  instrumented  test 
article  will  be  deployed  from  the  shuttle  and 
accelerated  in: a  the  atmosphere  and  back  to 
: uw-earch  orbit  ‘or  shuttle  recovery. 


‘oe  rati  ons 


K.arth-nrhi  1 1  ng  operations  capability  will 
■ »»  enhanced  through  a  continued  .and  expanded 
erf  art  in  automation  and  robotics.  The 
«hi«»etive  of  this  p  ogram  is  to  exploit  the 
potential  of  artificial  intelligence  and 
te  I erobr*r  i cs  to  increase  the  capability, 
flexibility,  and  safety  of  space-  and 
ground-operations  while  decreasing  associated 
costs.  The  NCOS  recommended  a  ’’special 
emphasis  on  intelligent  autonomous  systems.’ 
This  CSTI  element  responds  directly  to  the 
NCOS  recommended  acceleration  of  work  to 
support  complex,  automated,  remote  operations. 
Research  in  autonomous  systems  to  enable  the 
control  of  multiple  subsystems,  including  the 
capability  for  reasoning  and  recovery  from 
failures,  will  be  demonstrated  with  the 
technology  flowing  into  the  Space  Station  test 
beds.  Launch  system  software  development  will 
also  be  addressed  to  begin  replacing  as  muc- 
of  the  routine,  labor  intensivp  activities  as 
possible. 

A  robotics  program  will  demonstrate  a 
multi-arm.  highly  autonomous  capability  for 
remote  assembly,  repa: .  and  servicing  of  space 
vehicles.  This  program  is  built  upon 
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research  and  technology  development  In  five 
core  areas:  sensing  and  perception,  control 
execution,  task  planning  and  reasoning, 
operator  interface,  and  system  archl tecture 
and  Integration. 

Also  In  earth-orbltlng  operations  is  the 
requirement  for  high  capacity  power  systems. 
(>ngoing  research  will  continue  on  the  high 
power  Stirling  cycle  dynamic 
thermal-to-e lectric  power  conversion  system. 
Per f ormance  capabilities  are  aimed  at  a 
five-fold  increase  in  the  amount  of  electric 
power  that  can  be  generated  from  a  reactor  of 
the  size  being  developed  In  the  SP-100 
program  and  are  aimed  at  a  doubling  of  the 
power-to-weight  ratio. 

Science 

In  science  supporting  technology,  emphasis 
will  be  on  an  augmented  large  flexible 
structures  and  control  program,  and  on 
sensors  and  data.  Future  space  missions  will 
require  very  light,  very  large,  flexible 
structures  for  large  communications 
satellites,  science  observing  platforms, 
advanced  Space  Station  configurations,  a 
large  deployable  reflector,  and  large 
baseline  interferometers.  The  structural 
dynamics  and  control  response  of  such 
flexible  structures  is  net  well  understood. 

An  added  problem  is  the  degree  of  precision 
attainable  for  larg?  space-based  reflectors 
especially  in  the  submi ilimeter  range  for 
astronomical  observation.  To  address  this 
problem,  lightweight  reflector  panels  made  up 
of  composite  materials  tailored  for  9pace 
durability  and  surface  accuracy  are  being 
developed.  Active  controls  research  will  be 
included  as  an  approach  to  maintain  the 
surface  precision  and  precise  alignment  of 
seven  l-to-2  meter  wide  assembled 
panels . 

The  program  in  sensor  devices  will  be 
aimed  at  aubmillimecer  technology  for 
high-resolutlon  earth  science  and 
astrophysics  instruments  at  wavelengths 
outside  the  limits  of  current  observations; 
solid-state,  tunable  lasers  for  lidar 
instruments  to  sense  atmospheric 
constituents;  and  long-life  sub-Kelvin 
detector  cooler  technology  to  cool  infra-red 
detector  arrays  necessary  for  imaging  in  a 
low-thermal  background  such  as  that  found 
looking  out  Into  deep  space. 

In  the  data  area,  research  will  be 
augmented  on  high  rate,  high  capacity 
system  that  function  as  smart,  on-board 
processors  primarily  tor  imaging  information. 
A  combination  of  high  speed,  special  purpose 
processors  and  high  capacity  data  devices  is 


required  if  we  are  to  discontinue  excessive 
demands  on  ground  communications.  CSTI  data 
research  also  includes  verification  of  a 
read/ write  high  capacity  (a  trillion  bits) 
optical  disk  recorder.  This  unit  would  be 
useful  for  storing  images  for  overlays  and 
compar 1  sons . 

THE  SECOND  STEP 

If  CSTI  Is  characterized  as  a 
t  r  ansportat  ioi. ,  earth  orbit,  and  science 
supporting  focus,  then  the  second  increment, 
which  we  call  Pathfinder,  looks  beyond  Space 
Station,  beyond  LEO,  and  addresses  common 
technologies  that  support  a  range  of  future 
missions  Including  a  return  to  the  Moon, 
missions  to  Mars,  and  expanded  exploration  of 
the  outer  planets.  Pathfinder  includes 
technology  thrusts  to  enable  long-term 
human  presence  in  space,  processing  of 
non-terrestrial  materials,  an  autonomous 
rover,  interplanetary  t ranaportat i on , 
aerobraking  and  atmospheric  capture  for 
planetary  return,  precision  aero-* ecover / , 
tether  applications,  and  on-orLit  propellant 
depots  for  manufacturing  and  storage  of 
propellants. 

A  HEW  BEGINNING 

A  new,  Agency-wide  attitude  toward  research 
and  technology  development  is  emerging.  The 
oast  year  has  been  a  year  of  reflecting  upon 
the  state  of  NASA  and  reconstituting  the 
Shuttle.  Ve  should  learn  from  it,  be  stronger 
for  it,  but  let  it  be  a  turning  point  for  a 
renewed  dedication  to  the  civil  space  program 
—  a  new  beginning.  CSTI  and  Pathfinder  are 
right  for  NASA  and  the  Nation,  and  these 
efforts  will  challenge  and  encourage  the 
Agency,  industry,  and  the  University 
community.  And,  for  universities  in  the  space 
engineering  disciplines,  an  entirely  new  and 
important  role  may  be  emerging  in  the  scope 
and  manner  in  which  NASA  creates  university 
centers  for  space  technology  research. 

The  Nation's  technology  program  fortells 
what  we  will  be  doing  twenty  years  from  now. 

If  we  follow  through  on  what  we  have  started, 
the  picture  of  the  21st  Century  has  certainly 
become  brighter. 
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Computer  Architecture  for  Future  Spacecraft 


Jack  !3  Dennis 

Laboratory  for  Computer  Science 
Massachusetts  Institute  of  Technology 
o-fS  Technology  Square.  Cambridge,  MA  02139 


In  the  design  of  computers  for  spacecraft  there  is 
extraordinary  emphasis  on  low  weight,  small  size,  and 
minimum  power  consumption  These  considerations  mandate 
refined  packaging  and  cooling  technology,  and  use  of  a  logic 
technology  that  achieves  the  highest  computation  rate  for 
given  device  density  and  power  dissipation — consistent  with 
survivability  in  space  The  goals  have  been  more  than 
sufficiently  met.  and  computers  have  made  possible  many 
impressive  achievements  in  space. 

Advances  in  memory  and  logic  devices  have  led  people  to 
envision  that  much  higher  levels  of  function  and  intelligence 
can  be  built  into  future  spacecraft  within  the  limits  on  weight, 
size  and  power  Authors  foresee  autonomous  vehicles  guided  by 
knowledge-based  software  systems  and  elaborate  vision 
systems  The  motivation  is  to  make  the  best  use  of  the  tiny 
signal  bandwidth  available  for  communication  with  mother 
»arth. 

Nevertheless,  serious  problems  must  be  addressed  if  this 
potential  is  to  yield  practical  achievements  in  space,  especially 
where  the  advanced  computing  capability  is  to  be  applied  to 
mission  critical  or  real  time  tasks 

Dependability:  One  problem  area  is  dependability 
Significant  reliability  goals  have  been  met  in  past  space 
missions  spacecraft  have  survived  deep  space  missions 
requiring  dependable  performance  for  m any  years — a  truly 
remarkable  feat  This  has  been  accomplished  through  the  use 
of  redundancy,  fault  detection,  and  self  repair  using  standby 
spare  components  to  make  space  computers  tolerant  of  any 
single  point  failure 

The  needed  dependability  has  been  feasible  because 
software  functions  are  kept  simple  and  critical  software 


components  are  subjected  to  unusually  thorough  checking  and 
verification  during  development  Thus  thoroughness  and  the 
special  requirements  often  imposed  on  software  design  by  fault 
tolerance  schemes  place  severe  limits  on  the  complexity  of 
software  functions  that  may  be  undertaken  As  tasks  are  given 
to  the  spacecraft  computer  that  require  more  sophisticated 
software  structures,  the  difficulty  of  achieving  the  needed 
dependability  will  increase. 

Programmability:  The  most  significant  area,  however,  is 
programmability  While  present  architectures  for  spacecraft 
computers  may  be  a  reasonable  choice  for  the  software 
requirements  of  present  space  missions,  they  are  a  poor  match 
to  those  foreseen  in  the  future.  Successful  application  of 
artificial  intelligence  methods  and  the  effective  development  of 
reusable  software  parts  will  require  significant  architectural 
innovation  if  the  potential  of  these  techniques  is  to  be  realized 
in  the  construction  of  very  large  software  systems  for  the  space 
environment. 

One  requirement  of  the  knowledge-based  approached  to 
implementing  intelligent  behavior  is  a  large  memory  If  the 
memory  of  a  spacecraft  computer  is  entirely  in  the  form  of 
static  semiconductor  devices,  then  3ize  and  power  limitations 
will  severely  limit  the  level  of  intelligent  behavior  possible 
The  principal  technologies  available  for  large  spacecraft 
memories  are  dynamic  semiconductor  devices  and  magnetic 
bubble  memories.  High-capacity  dynamic  memories  are  very 
susceptible  to  soft  errors  and  other  forms  of  failure  from 
radiation  exposure,  and  they  have  a  relatively  slow  access  time 
Bubble  memories,  while  resistant  to  radiation,  have  such  long 
access  times  that  a  hierarchical  memory  organization  will  be 
required. 
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These  facts  complicate  the  design  of  computers  having 
storage  systems  to  support  modular,  knowledge- based 
software  6,  7  3,  1  This  is  because  the  advanced  software 
iesired  requires  automatic  management  of  the  memory  system 
including  recognition  and  recovery  of  storage  occupied  by  data 
no  longer  of  relevance  The  schemes  adopted  for  fault 
detection,  self  repair  and  software  recovery  must  be  consistent 
with  these  requirements.  Specifically,  if  the  fault  tolerance 
schemes  are  not  transparent  to  the  sensitive  application 
software,  then  these  schemes  will  impose  intolerable 
restrictions  on  the  software  design.  1  am  not  personally  aware 
of  work  that  addresses  these  issues  in  a  sufficiently  general 
way  The  achievement  of  fault  tolerance  in  an  advanced 
programming  environment  is  an  unmet  challenge  to  computer 
system  architects 

Dataflow  Technology  and  it»  Role  in  Space:  Present 
and  future  spacecraft  computers  will  continue  to  have 
redundant  elements  to  meet  dependability  requirements  They 
will  also  be  built  of  multiple  processing  elements  to  obtain  the 
most  performance  for  given  weight,  size  and  power  In  a 
dataflow  computer  each  processing  element  implements  a  data 
driven  instruction  execution  mechanism  jlj. 

Weight.  Size  and  Power  Studies  „f  applications  show  that 
data  flow  computers  have  peak  performance  capabilities  for 
given  weight,  size  and  power  well  beyond  that  of  other 
multiprocessor  architectures  8.  4j.  Moreover  these  same  studies 
indicate  that  this  peak  performance  can  be  closely  approached 
in  many  important  application  areas.  Thus  dataflow  computer 
architecture  should  be  seriously  considered  for  spacecraft 
applications 

Programmability.  The  program  structures  used  with 
dataflow  machines  are  nicely  matched  to  the  phrases  used  to 
express  computations  in  high  level  functional  programming 
languages  Hence  there  is  a  natural  way  to  split  programs  into 
modules  that  make  sense  at  both  the  language  level  and  the 
machine  level 

We  have  recently  found  that  the  basic  artificial  intelligence 
programming  scheme— heuristic  search  of  a  tree-structured 
space  of  possibilities — can  be  implemented  with  extraordinary 
efficiency  on  dataflow  computers  of  relatively  simple 
architecture  51.  Thus  dataflow  computers  offer  an  attractive 
basis  for  supporting  advanced  software  for  space  missions. 


that  is  transparent  to  the  application  software  10  For  uher 
types  of  <-omputer  systems,  this  has  been  achieved  rarely  m  i 
only  for  restricted  classes  of  high  level  language  programs  The 
expectation  of  machines— dataflow  computers  in  particular 
that  directly  implement  powerful  functional  languages  makes 
the  vision  of  space  dependable  computers  performing 
sophisticated  intelligent  procedures  a  realistic  vision  for  'he 
future  9 
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